Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents

VAKRA Dataset | LeaderBoard | Release Blog | GitHub | Submit to Leaderboard

We recently introduced VAKRA, a tool-grounded, executable benchmark for evaluating how well AI agents reason and act in enterprise-like environments.

Unlike traditional benchmarks that test isolated skills, VAKRA measures compositional reasoning across APIs and documents, using full execution traces to assess whether agents can reliably complete multi-step workflows.

VAKRA provides an executable environment where agents interact with over 8,000+ locally hosted APIs backed by real databases spanning 62 domains, along with domain-aligned document collections. Tasks can require 3-7 step reasoning chains that combine structured API interaction with unstructured retrieval under natural-language tool-use constraints.

As can be seen below, models perform poorly on VAKRA - in this blog, we include additional dataset details about the tasks in VAKRA and present an analysis of failure modes we observed on different tasks.

Task Description

As shown below, the VAKRA benchmark comprises of four tasks, each testing a different set of capabilities.

Fig 1: Representative examples of each capability in the VAKRA benchmark

Capability 1: API Chaining using Business Intelligence APIs

This capability includes 2,077 test instances across 54 domains, requiring the use of tools from the SLOT-BIRD and SEL-BIRD collections (Elder et al., 2026). Compared to the setup in Elder et al., the tool universe in SLOT-BIRD and SEL-BIRD is expanded through the inclusion of a larger number of domains. Each domain is restricted to one tool collection, and tasks involve chaining 1–12 tool calls to arrive at the final answer.

Fig 2: Data sample from SEL-BIRD collection

Fig 3: Data preview obtained from `get_data` function

Capability 2: Tool Selection using Dashboard APIs

Capability 3: Multi-Hop Reasoning using Dashboard APIs

Capability 4: Multi-Hop, Multi-Source Reasoning and Policy Adherence

Evaluation Framework

Evaluation Metric

Scoring