What Does the Language Model Game Benchmark Test?

Alex Zhang's benchmark on language models playing video games asks whether vision language models are good enough at plugging into games with the latency constraint included. The benchmark came out right after Claude Plays Pokemon. About two years later, he calls the release ancient.

The numbers are very outdated. A lot of the models are very different, and some folks are running newer models on the games. The fascination behind the work is whether an intelligent system can play new games. The game examples also sit with a separate point of excitement: a unique way to capture language and understanding in a very, very fast model.

The premise keeps the latency constraint inside the test of plugging vision language models into games. It is a check of whether the models are good enough under that constraint, and the original numbers no longer describe a lot of the models now in use.