(Credit: Virtual Protocol)
Generative AI can not only simulate a playable version of Doom, but also the Nintendo classic Super Mario Bros., according to new research.
On Wednesday, Virtual Protocols, a company that’s been developing AI-powered characters for games, debuted a research project called MarioVGG, which uses text-to-video generation to recreate a version of the 1985 Nintendo game. Its research paper concludes that AI-enabled video generation programs show real potential in providing a new way to develop games.
“Trained on just a single consumer-grade GPU (an Nvidia RTX 4090), our text-to-video diffusion model, MarioVGG can generate controllable game frames that are consistent and follow game mechanics,” the paper adds.
MarioVGG was developed by training the AI model on 737,000 frames taken from a Super Mario Bros. session. This included identifying frames where the main character Mario “runs” or “jumps.”
The model was then trained to generate copycat images of the Nintendo game based on input from the player, which arrives as a text prompt, either run or jump. “Specifically, our model takes an initial video frame of the game along with text of the desired action (e.g. ‘jump'), and learns to generate a sequence of frames that visually depicts the desired action,” the paper adds.
(Credit: Virtual Protocols)The result has some lag and is not easily playable. According to the paper, MarioVGG “takes approximately 4 seconds to generate a 6-frame video sequence of the game at inference.” Each frame is also generated at 64-by-48 or a 128-by-96 resolution when real Nintendo games from the era feature a 256-by-224 resolution. Even so, the AI model can accurately simulate the 2D Mario experience, including the various mechanics, like the real 1985 game.
“We observe that MarioVGG successfully simulates and generates the gravity mechanics of the game,” the paper says. “The generated video demonstrates Mario falling off a platform as the ‘Run’ action is used, all this while maintaining consistency of the surrounding environment and generating coherent new tiles.”
(Credit: Virtual Protocols)In addition, MarioVGG will generate its own unique Mario levels, with the first input frame the only one "grounded from a true game state."
But the model still faced some limitations. This included MarioVGG failing to always follow the expected behavior and rules of the game when generating the video frames. For example, "levels and objects in MarioVGG are not controllable and are generated procedurally," the paper says. Another issue is that the AI model will keep rendering video frames even after Mario dies, rather than revert to the last checkpoint.
That raises questions over how an AI-generated game will work like a traditional computer game, which can involve tracking the character’s stats and progress through a level. “An open research question is how terminal states are decided in a fully generative game that has no ‘ground truth’ equivalent,” the paper says.
Still, Virtual Protocols says its research offers a glimpse at future applications of generative AI for game development, pointing to how MarioVGG only used a relatively small sample of training data and one consumer-grade GPU.
“The era of infinite interactive worlds is here,” the company wrote in a tweet.