Large language models (LLMs) have demonstrated impressive progress in recent years, prompting the development of increasingly complex benchmarks to truly test their capabilities. However, their advancements haven't been uniform across all areas. One area where they consistently fall short is video games.

While there have been isolated instances of AI achieving victory in specific games – for example, one model purportedly beat Pokemon Blue some time ago – these successes are more the exception than the rule. Even when an AI manages to complete a game, it often does so at a significantly slower pace than a human player. Furthermore, its gameplay is frequently characterized by peculiar, repetitive errors, and often necessitates specialized software to facilitate interaction with the game environment.

Julian Togelius, director of New York University’s Game Innovation Lab and co-founder of Modl.ai, a company specializing in AI game testing, recently delved into the implications of LLMs' shortcomings in the realm of video games. In a research paper, Togelius explored what this apparent deficiency reveals about the broader state of artificial intelligence. He discussed how the limitations of LLMs in gaming provide valuable insights into the current capabilities and boundaries of AI.

One key factor contributing to this struggle is the nature of video games themselves. They often require a combination of skills that LLMs haven’t yet mastered, including real-time decision-making, spatial reasoning, and the ability to adapt to unpredictable situations. Unlike tasks involving static text or data, video games present a dynamic and constantly changing environment. LLMs, which are primarily trained on vast amounts of text data, struggle to generalize their knowledge to these interactive scenarios.

Moreover, video games often involve a degree of creativity and intuition that is difficult to replicate with current AI technology. Human players can draw upon their past experiences and understanding of the game world to develop novel strategies and overcome challenges. LLMs, on the other hand, tend to rely on pre-programmed algorithms and statistical patterns, which can limit their ability to think outside the box.

The difficulties LLMs face in mastering video games highlight the limitations of current AI approaches. While they excel at tasks involving pattern recognition and data analysis, they still struggle with tasks that require real-world understanding, adaptability, and creative problem-solving. This suggests that further research is needed to develop AI systems that are more capable of handling complex, dynamic environments like those found in video games. The inability of LLMs to consistently perform well in video games serves as a reminder that AI, despite its rapid advancements, still has a long way to go before it can truly replicate human intelligence.