These Researchers Made AI Drive a Toyota Corolla to Get In-N-Out

Aditya Ramabadran, Simon Mahns, and Tobias Gessler, three AI engineers at a startup called Axiom, recently fancied In-N-Out Burger for lunch.

They weren’t about to drive themselves, though. Sitting in a 2024 Toyota Corolla near the Bay Area restaurant’s drive-thru lane, they opened a laptop and asked OpenAI’s GPT-6 Astra to take the wheel. They linked a chat interface to a server, which was connected to several windscreen-mounted cameras and the car’s power steering system. A safety driver kept one foot above the brake, just in case.

The AI model, which is normally tasked with generating text, code, and the odd image, slowly but surely navigated the vehicle up to the take-out window so they could collect their food.

“Maybe AGI is here after all,” one of the engineers remarked, referring to artificial general intelligence, the much-ballyhooed idea of machines that can match human intelligence.

Self-driving cars are nothing new, of course, but they’re normally operated by algorithms specifically trained and engineered for the task. In this case, the operation came from a model designed to output text, and it happened on the fly, with no prior coaching by the trio. The fast-food stunt, as well as a number of other experiments, suggest that language-based AI models are starting to gain a rudimentary but useful understanding of the physical world.

Though nothing overtly alarming happened during the trio’s lunchtime jaunt, they admit that putting a general-purpose model in charge of a fast-moving, two-ton hunk of steel is a high-stakes undertaking. As physical understanding improves, we could see AI models reach into the real world in impressive—and perhaps dangerous—new ways.

Getting Physical

Elorian and Scale AI, a company that provides training data to big AI labs, recently developed a new benchmark, Humanity’s Sixth Sense, which measures models’ ability to understand physical scenes.

This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.