CodeAgents + Structure: A Better Way to Execute Actions

Today we're sharing research that bridges two powerful paradigms in AI agent design: the expressiveness of code-based actions and the reliability of structured generation. Our findings show that forcing CodeAgents to generate both thoughts and code in a structured JSON format can significantly outperform traditional approaches across multiple benchmarks.

Figure 1: Accuracy comparison of three approaches: Structured CodeAgent (blue), CodeAgent (orange), and ToolCallingAgent (gray) on SmolBench (GAIA, MATH, SimpleQA, and Frames). Error bars represent 95% Confidence Intervals.

🤔 The Evolution of Agent Actions

AI agents need to take actions in the world - whether that's calling APIs, processing data, or reasoning through complex problems. How agents express these actions has evolved through several paradigms:

Traditional JSON Agent: Agents generate structured JSON to call tools.

{"tool": "get_weather", "arguments": {"city": "Paris"}}

These agents operate by selecting from a list of predefined tools and generating JSON-formatted calls. This method for calling tools has been popularized by OpenAI's function calling API, and has since then been the most widely used method to call tools.

It is reliable, but limited by:

• A limited set of actions: The actions the agent can take are expressed only through predefined tools which limit its functionality.
• Lack of composability: If the task requires composing information from multiple sources, JSON agents struggle because they lack support for maintaining intermediate state across tool calls. While some models support parallel tool calls, they can't easily handle scenarios where one tool's output determines the next action or where results need to be compared and processed together.
• Rigid structure: Very limited in handling cases where tools do not match exactly what needs to be done.

Code Agents: Agents make use of their innate coding ability and write executable Python code directly.

Pros of using a CodeAgent: