(Credit: Google Research)
We’ve all seen how AI image generators can churn out pictures of whatever you’d like. But what if you took the same technology and applied it to generating stills for a playable game?
Researchers at Google recently used this concept to develop an AI model that’s capable of simulating the 1993 classic PC shooter Doom — but without using computer code from the game itself. Instead, the researchers' model works by pumping out stills for the game like an AI image generator does, except it can do so in real-time at over 20 frames per second for a playable experience.
The model is called GameNGen, and it’s the subject of a new paper from researchers at Google and Tel Aviv University. “Can a neural model running in real-time simulate a complex game at high quality? In this work we demonstrate that the answer is yes,” they write. “Specifically, we show that a complex video game, the iconic game Doom, can be run on a neural network.”
In the paper, the researchers note a computer game fundamentally works like this: the player makes an action or input, the game state updates accordingly, and then it renders the result on the screen. This so-called “game loop” creates the illusion that you’re in an interactive virtual world, even though your computer is just showing you changing pictures on the screen.
The researchers used Stable Diffusion version 1.4, an open-source AI image generator. They also developed a separate AI model to play the real Doom game while recording the footage for a total of 900 million frames. The resulting training data is then used by Stable Diffusion to pump out game images, adapting them as it receives inputs from the player.
(Credit: Google Research/Tel Aviv University)The team posted several clips of GameNGen rendering Doom, including footage of human players trying it out. The results show the AI model is able to accurately simulate the classic PC shooter both visually and on a gameplay level. For example, the model can simulate a door opening as the player approaches and a fireball hitting the player, taking away some health.
However, GameNGen also contains some major limitations. “The model only has access to a little over 3 seconds of history,” the researchers wrote. As a result, enemies and objects can sometimes pop in of nowhere and then disappear seconds later. Nevertheless, GameNGen is able to create the illusion it can remember the game world because each rendered image allows the model to infer the player’s ammo, health status, weapons, and location.
The other issue is that a traditional computer game can be quite complex. In addition to rendering pixels on a screen, a game can contain dialogue, numerous characters, along with story and game mechanics that can happen off-screen. But despite the limitations, the researchers say GameNGen shows how generative AI could transform game development, potentially leading to AI-created games, which Nvidia's CEO has also predicted could occur in the next five to 10 years.
“For example, we might be able to convert a set of frames into a new playable level or create a new character just based on example images, without having to author code,” the researchers wrote in their paper while adding: “Today, video games are programmed by humans. GameNGen is a proof-of-concept for one part of a new paradigm where games are weights of a neural model, not lines of code.”