Tremble, Mario: This AI is capable of inventing platforming video games from scratch

The latest demonstration by OpenAI of Her amazing generative model Sora Boundaries showed Convert text to video. Now, Google DeepMind brings these capabilities to life. Convert text into video games.

The new model, called Genie, can take a brief description, hand-drawn diagram or photo and turn it into a video game in the style of classic 2D platformers. Like Super Mario Bros. But don't expect anything quickly. Games run at 1 frame per second, compared to the 30 to 60 frames per second typical of most modern games.

“it's a great job,” says Matthew Guzdial, an artificial intelligence researcher at the University of Alberta, who developed a similar game generator a few years ago.

Jenny was trained on 30,000 hours of video from hundreds of 2D platform games scraped from the Internet. According to Guzdial, others have already taken this approach. His game creator learned from videos to create abstract platform games. Nvidia used video data to train a model called GameGAN, capable of producing versions of games like Pac-Man.

But all of these examples trained the model using input actions (such as pressing buttons on a controller) as well as video sequences: a video frame showing Mario jumping was paired with a jumping action, and so on. Classifying video sequences with input actions requires a lot of work, which has limited the amount of training data available.

instead ofJenny was trained using video sequences only. Next, you learned which of eight possible actions would cause a video game character to change position. In this way, countless hours of online video become potential training data.

Genie can create simple games from hand-drawn graphics. /Google DeepMind

The genie Create each new game frame on the fly Depending on the action the player takes. If you press Jump, Genie will update the image to show the game character jumping; If you press Left, the image will change to show the character moving to the left. The game progresses through action and each new frame is created from scratch as the player plays.

Future versions of Genie may run faster. “There are no fundamental constraints preventing us from achieving this 30 fps“Genie uses many of the same techniques used by large contemporary language models, where significant advances have been made in improving the speed of inference,” says Tim Rocktachel, a Google DeepMind research scientist who leads the team behind the work.

Jenny learned some visual quirks common in platform games. Many games of this type use the parallax effect, where the foreground moves horizontally faster than the background. Genie often adds this effect to the games he creates.

Although Genie is an internal research project and will not be put up for sale, Guzdial notes that the Google DeepMind team says it could one day become a game creation tool, something it's also working on. “I'm very interested in seeing what they build,” he says.

Virtual playgrounds

But researchers at Google DeepMind are interested in more than just creating games. The team behind Genie is working on open learning, where AI-controlled robots are dropped into a virtual environment and left to solve various tasks by trial and error (a technique known as reinforcement learning).

In 2021, another DeepMind team developed a Virtual playground It's called X-Land Where I learned robotics Collaborate on simple tasksHow to move obstacles. This type of Sandboxes They will have a crucial role in training future robots in a series of different challenges before confronting them with real-world scenarios. Video game examples demonstrate that Genie can be used to create these virtual playgrounds.

Others have developed similar tools. For example, David Ha of Google Brain and Jürgen Schmidhuber of the IDSIA AI Lab in Switzerland developed a tool in 2018 to train… Robots In game-based virtual environments called Models of the world. But again, unlike Genie, this required that the training data include input actions.

The team showed how this capability is also useful in robotics. When Jenny was shown videos of real robotic arms manipulating various household objects, the model learned what actions the arm could perform and how to control it. Future robots can learn new tasks Watch video lessons.

“It's hard to predict what use cases will be possible,” Rocktachel says. “We hope that projects like Genie will end up providing people with it New tools To express their creativity.