Key Points

1. GameNGen is the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality.

2. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU.

3. Next frame prediction in GameNGen achieves a PSNR of 29.4, comparable to lossy JPEG compression.

4. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the GameNGen simulation.

5. GameNGen is trained in two phases: (1) an RL agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions.

6. Conditioning augmentations enable stable auto-regressive generation over long trajectories in GameNGen.

7. The pre-trained Stable Diffusion v1.4 auto-encoder is fine-tuned to improve image quality in GameNGen.

8. GameNGen only requires 4 DDIM sampling steps to generate high quality frames, much fewer than typical diffusion models.

9. Noise augmentation is crucial to prevent auto-regressive drift and maintain stable generation over long trajectories in GameNGen

Summary

The research paper introduces GameNGen, the first game engine powered entirely by a neural model. The engine can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU, achieving a high-quality visual output. The training process involves two phases: an RL-agent learns to play the game and record training sessions, and a diffusion model is trained to produce the next frame based on the sequence of past frames and actions. Conditioning augmentations enable stable autoregressive generation over long trajectories.

The authors highlight that computer games are manually crafted software systems involving game loops, which gather user inputs, update the game state, and render it to screen pixels. While there are diverse game engines, the game state updates and rendering logic are handcrafted in all cases. The paper identifies the need for a neural model that can simulate interactive worlds of video games and outlines the limitations of existing approaches to simulate complex games at high quality, speed, stability, and in real-time.

GameNGen addresses these limitations, demonstrating that a neural model can simulate a complex video game like DOOM in real-time with high quality. The engine is trained using a generative diffusion model conditioned on the RL-agent's trajectories. The research explores techniques to mitigate auto-regressive drift using noise augmentation and a pre-trained auto-encoder for better image quality. The study also evaluates the capabilities of the engine, demonstrating that it achieves a high level of simulation quality comparable to the original game and that human raters struggle to distinguish between the simulation and the actual game.

The authors acknowledge the limitations of GameNGen, such as limited memory and differences in the agent's behavior compared to human players, and propose future work to address these limitations. They also discuss the potential of GameNGen as a step towards a new paradigm for interactive video games, where games are automatically generated by neural models and the possibilities for development and modification of games facilitated by this new paradigm.

The paper provides detailed experiments, training processes, and evaluation metrics to demonstrate the capabilities and limitations of GameNGen, positioning it as a substantial contribution to the field of game engine development powered by neural models.

In summary, the research paper presents a novel game engine, GameNGen, powered entirely by a neural model, which can interactively simulate the classic game DOOM at high speed and quality, pushing the boundaries of real-time, high-quality game simulation using neural models. It outlines the training process, the technical challenges, and the evaluation results, demonstrating the efficacy and potential of this new approach to game engine development.

Business Listing: https://arxiv.org/abs/2408.148...