The Next Evolution of AI Is Learning From Your Dodgy Gaming Skills

With the twirl of a thumbstick, squeeze of a trigger, and press of a few buttons, even an unskilled player can dance their way through a 3D video game environment. A British startup is wagering that straightforward sequences like these contain a trove of information that can be used to train new artificial intelligence models.

There is a growing belief in corners of the AI industry that large language models will ultimately be limited by their inability to navigate the physical world. Trained only on words, LLMs are perhaps ill-equipped to pilot autonomous vehicles, steer robotic arms, or perform any other action that requires finesse and precision. To remedy that shortcoming, a crop of celebrated researchers including Fei-Fei Li and Yann LeCun are focusing their efforts on a different class of AI: world models.

To become fluent in real-world physics, world models need to be trained on a combination of visual and action data. Before it can deftly maneuver a robotic arm, a model might need to be trained on video footage of the factory floor, coupled with information about how firmly an object should be gripped, with what torque it’s manipulated, and so on. But unlike LLMs, trained on oceans of text, there is no similar corpus of material available to world model labs.

“For world models, you need cause and consequence,” says Xiatian Zhu, an associate professor specializing in AI at the University of Surrey. “On the internet, we have very little of this type of data.”

“There are millions of great games, and they are more and more similar to the real world,” Rhea Loucas, CEO of Worldmodeldata, tells WIRED. “Why don’t we take the vast, abundant, diverse experiences from video games, and teach AI?”

Some labs have tried to manually generate their own data by attaching sensors to humans and robots in testing environments. But that approach yields only a small amount of data, and it doesn’t account for the fringe scenarios a model might encounter in the wild.