AI agents have bad taste in long-term decisions and a new benchmark proves it

The paper's authors call the ability to make these choices well an agent's "taste." Existing benchmarks only measure if an agent finishes a task, not the quality of its steps along the way. Taste-Bench is designed to fill that gap.

Long-term decisions can't be judged in the moment

You can't judge a long-term choice right away. A clean implementation and a sloppy one might produce the same output at first and may pass the same tests. The clean approach only pays off when later changes come more easily. Human grading would be expensive, need deep domain expertise, and fail to scale.

So the researchers rely on hindsight. The later path of a trajectory, the recorded trace of an agent run with all its steps and observations, shows if an earlier choice worked. In practice, agent systems try the same task multiple times. Those attempts eventually split off in different directions.

Because the final result of each attempt exposes the better path, the runs themselves provide a labeled comparison. The authors call this branching point a "decision fork." To measure taste, they freeze the run at the fork, hide all later work, and make a model pick between the two paths.

Past mistakes reveal where agents go wrong

Taste-Bench contains 502 questions, each with the original task, the trajectory up to the fork, and two possible directions. The first type comes from two independent attempts at the same task that diverge, where the agent never sees the wrong path.

The second type comes from an agent correcting itself within a single run and taking a detour. Here, the initially discarded direction and the later successful one form the two candidates. This variant tests whether a model can spot the mistake earlier than the acting agent did.