The Reward Problem: Why AI Agents Go Rogue

The reinforcement-learning pioneer says the hard part has always been choosing the reward, and this summer’s agent break-ins show why.

The OpenAI agents that broke into Hugging Face this summer were chasing a goal they had been given. But they did so by means nobody intended. Since then, Google, Anthropic and Meta have disclosed similar incidents, leading to public anxiety about rogue AI.

To better understand what’s happening, I went to see Andy Barto. He and Richard Sutton wrote the standard textbook on reinforcement learning, and they shared the 2024 Turing Award for laying the foundations of the field. OpenAI blamed the Hugging Face break-in largely on “reward hacking,” when an AI system fixates on getting a reward rather than the goal its designers intended. In its report on the incident, the company said that during training with reinforcement learning, the agents were occasionally rewarded for cheating on tasks they couldn’t otherwise solve.

Getting the reward right is the problem, Barto believes. Reinforcement-learning systems chase rewards they are given, but nobody has found a reliable way to write one that captures what the designers actually want.

A Visit to Cape Cod

Barto is retired from the University of Massachusetts Amherst and spends much of his time on Cape Cod. I found him sitting beside sliding glass doors that give onto a wooden deck overlooking one of the Cape’s famous kettle ponds. The ponds sit where a block of glacial ice slowly melted, leaving the shape of the ice long after the ice is gone. A language model is something like that. It learns by predicting text from an enormous body of writing. It doesn’t keep most of the writing. But it keeps the shape.

Much of how language models behave is shaped in post-training by reinforcement learning. The model gets a score meant to reflect how well it did what the designers want, and it learns to push that score up. That shapes how it plans and how long it keeps trying.