Google Unveils GameNGen: Integrating Generative AI into Gaming

On 27th August 2024, Google released a paper detailing the capabilities of their new model GameNGen. This research was authored by Dani Valevski (Researcher at Google Research), Yaniv Leviathan (Engineer at Google Research), Moab Arar (PhD candidate at Tel Aviv University), and Shlomi Fruchter (Engineer at Google DeepMind). 

GameNGen’s Capabilities 

Powered entirely by a neural model, GameNGen is one of the pioneer search engines that allows high-quality, real-time interactions with complex environments over long periods. “GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression”, per Google. The training occurs primarily in two phases:  (1) an RL agent learns to play the game, with training sessions being recorded, and (2) a diffusion model is trained to predict the next frame using the sequence of previous frames and actions. 

This approach, while still in its nascent stage, is testament of a new paradigm where games are weights of a neural model, not lines of code, in an era where video games are programmed by humans. GameNGen reveals that a neural model can interactively run a complex game like DOOM on existing hardware. 

Architecture & How It Works

In order to collect data efficiently, an RL agent plays the game, and its actions and observations are recorded as training data. For more enhancement and consistency, the Stable Diffusion model is trained to predict the next game frame using previous frames and actions. Lastly, the model's decoder is fine-tuned to improve image quality and reduce visual artefacts. 

AI in Gaming

Jack O Brien, ex Google, remarked on how this research is seminal and opens up new possibilities for the world of video generation and the use of AI.  Jensen Huang, Founder of NVIDIA, has been bullish on the inevitability of the world of gaming and the role played by AI in making it a reality. “We used AI to revolutionise computer graphics. Graphics enabled AI, and now AI is saving graphics", Huang said. Last year, OpenAI’s acquisition of Global Illumination, a gaming and design startup and the creators of Biomes, highlights OpenAIs gaming focus for AI simulation and use. As AI research advances, companies will increasingly test their models in game worlds before real-world deployment.