Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore

General-purpose agents handle a broad range of tasks, but you still need them to follow the procedures that run your business: compliance checks, document-processing workflows, escalation policies, engineering conventions. Encoding all of that in one system prompt or in application logic gets hard to maintain and update. Skills are a modular alternative. A skill is a reusable set of instructions, usually stored in a SKILL.md file, that teaches an agent a domain-specific task like redacting a contract, reconciling an invoice, or following a team’s pull-request conventions. Because skills follow the open Agent Skills standard, they are portable across compatible harnesses, and the agent loads only the skill it needs at runtime instead of carrying every procedure in its core instructions.

A skill packages one or more tools with the context an agent needs to use them correctly:

• Instructions: Domain-specific guidance and constraints injected into the agent’s context.
• Tool bindings: The APIs, Model Context Protocol (MCP) servers, or local commands the skill depends on.
• Knowledge: Reference material and worked examples.
• Workflow: The multi-step procedure or decision logic the skill follows.
• Guardrails: Format requirements, scope limits, and validation rules.

This modular approach helps teams specialize agents faster, reuse proven procedures across agents and workflows, keep behavior consistent, and update domain-specific guidance without fine-tuning the underlying model or rewriting the agent’s core logic.

This skill composability in agents introduces two failure modes that general output-quality metrics can miss: the agent invokes a skill that is not appropriate for the task and the agent invokes the right skill but skips or only partially follows its instructions. Both failures can produce a fluent, plausible response without having used your pre-determined domain knowledge. An evaluation therefore cannot examine the final response alone.

To make these failures measurable, Strands Evals SDK and Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, add skill-focused evaluators:

• Skill Selection Accuracy determines whether each invoked skill was an appropriate choice for the task. It returns a binary result for each invoked skill.
• Skill Instruction Following determines how fully the agent followed an invoked skill’s instructions. It returns a five-level rating grounded in evidence for each prescribed step.
• Additionally on Strands Evals, Skill Invoked is a deterministic check of whether a named skill has been loaded successfully.

In this post, you will learn how to evaluate skill selection and instruction following from a recorded trajectory in Strands Evals, add deterministic routing checks to a test suite, evaluate skill behavior from OpenTelemetry traces with AgentCore Evaluations, and interpret per-skill results to choose the right fix, all through the AgentCore CLI.

Understand what each evaluator measures

An agent receives a task and a catalog of skills, chooses a skill, loads it, and acts. The run is recorded as a trajectory in Strands Evals or an OpenTelemetry trace in your observability layer. This record can now be used for all three skill evaluators. Skill Selection Accuracy checks whether each invoked skill fits the task and whether the agent invoked the correct skill. The following figure shows how the agent chooses a skill from the 1:n skills provided to it. Skill Selection Accuracy then scores whether the selected skill is the correct one for the task. You can find the prompt template and the rubric of this evaluator in the prompt template documentation.

Figure 1: Skill Selection Accuracy checks whether the agent chose an appropriate skill for the task

Skill Instruction Following asks how fully the agent followed that skill’s prescribed steps. You can find the prompt template and the rubric for this evaluator in the prompt template documentation.

Figure 2: Skill Instruction Following measures how completely the agent followed the skill’s steps

SkillInvoked is deterministic. It calls no model and is specific to Strands Evals.

An overview of these skill evaluators is demonstrated diagrammatically in the following figure.

Figure 3: Overview of the three skill evaluators

Consider an HR assistant agent with skills for paid time off (PTO) planning and discussing employee benefits. An employee asks about their dental and vision benefits. If the agent invokes the benefits skill, it may produce a more polished response. If the tool call succeeded but the agent chose the wrong playbook, Skill Selection Accuracy isolates that routing decision.

Now suppose the agent correctly invokes the PTO-planning skill for a related request. The skill instructs the agent to identify the employee_id, check the PTO balance, check the rollover rules against the latest HR policy, and then submit a PTO request if the conditions allow. If the agent checks the PTO balance but skips the rollover rules, the agent might still return a plausible response while violating the prescribed process. Skill Instruction Following isolates that execution failure and identifies the skipped step.

The failures require different fixes. An inappropriate selection often points to overlapping or ambiguous skill descriptions. Incomplete instruction following might call for clearer steps, a different skill structure, or a more capable agent model.

Evaluator	Availability	Score	Question answered
Skill Selection Accuracy	Strands Evals and AgentCore Evaluations	Binary, per invoked skill	Was invoking this skill appropriate for the task?
Skill Instruction Following	Strands Evals and AgentCore Evaluations	Five levels, per invoked skill	How fully did the agent follow this skill’s prescribed steps?
Skill Invoked	Strands Evals	Binary, deterministic	Was this named skill successfully loaded?

Because judge-based evaluators return per-invoked-skill results, multi-skill runs remain diagnosable: you can identify which selection or instruction-following result lowered the aggregate score. If no skill is invoked, the judge-based evaluators don’t produce a score. Pair them with SkillInvoked when a regression test has a known routing requirement.

Prerequisites

• Python 3.10 or later.
• An AWS account with Amazon Bedrock access, and credentials with InvokeModel permission for the judge model.

To follow the Strands Evals section, install the SDKs:

pip install strands-agents-evals strands-agents

You also need a recorded agent run. The skill evaluators accept either a Strands Evals Session or a raw message list as the trajectory. At launch, skill extraction recognizes signals from the Strands AgentSkills plugin, Claude Code, Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK, and generic SKILL.md file reads.

To follow the AgentCore Evaluations section, you need:

• An agent hosted on Amazon Bedrock AgentCore runtime or elsewhere. We will use the example of the HR assistant agent which you can deploy in your account.
• Observability enabled for that agent, so it delivers telemetry to Amazon CloudWatch.
• Transaction Search enabled in CloudWatch.

The examples in this post use the AgentCore CLI:

npm install -g @aws/agentcore

Evaluate a recorded trajectory with Strands Evals

Strands Evals is useful when you control the test cases and can rerun the agent during development or continuous integration. Check out the complete Strands evals code sample created for the HR assistant agent in the complete code sample.

1. Define the case and evaluators

from strands_evals import Case, Experiment
from strands_evals.evaluators import (
    SkillInstructionFollowingEvaluator,
    SkillInvoked,
    SkillSelectionAccuracyEvaluator,
)

case = Case(
    name="q3-revenue-tables",
    input="Summarize the revenue tables in q3-report.pdf",
)

evaluators = [
    SkillSelectionAccuracyEvaluator(),
    SkillInstructionFollowingEvaluator(),
    SkillInvoked(skill_name="pdf-table-extraction"),
]

2. Run the agent and capture its trajectory

The skill evaluators read the run’s trajectory. TracedHandler collects the agent’s spans and attaches them as the trajectory:

from strands import Agent
from strands_evals import TracedHandler, eval_task

@eval_task(TracedHandler())
def task_function():
    return Agent(...) # your skill-equipped agent

experiment = Experiment(cases=[case], evaluators=evaluators)
report = experiment.run_evaluations(task_function)
report.run_display()

3. Interpret the report

Suppose the agent loaded pdf-table-extraction, ran pdftotext -layout, but never opened the extracted file or located the table boundaries. A simplified report:

SkillSelectionAccuracyEvaluator: score=1.00, pass=True
pdf-table-extraction: The skill directly matches the request.
SkillInstructionFollowingEvaluator: score=0.50, pass=False
pdf-table-extraction: The extraction phase was completed,
the table-boundary phase was skipped.
Steps:
- Extract text with layout preservation: covered
- Locate table boundaries: skipped
- Summarize each table's headline figure: partial
SkillInvoked: score=1.00, pass=True
skill 'pdf-table-extraction' was invoked

Skill Instruction Following uses five ratings: Fully Followed (1.0), Mostly Followed (0.75), Partially Followed (0.5), Minimally Followed (0.25), and Not Followed (0.0). It passes at Mostly Followed or better.

Turn the checks into a deployment gate

• Add SkillInvoked for every critical skill that a specific regression case must invoke. Because it doesn’t call a model, it is a fast routing assertion.
• Gate the build on report.test_passes: if not all(report.test_passes): raise SystemExit(1).
• Use aggregate scores to track broader trends, but calibrate the threshold on your own cases before enforcing it.

Evaluate production traces with AgentCore Evaluations

AgentCore Evaluations works directly with existing OpenTelemetry traces, supporting on-demand evaluation, batch processing of stored sessions, and continuous sampling of live traffic. Telemetry is organized into sessions, traces, and spans. Because skill evaluators operate at the tool-call level, each result includes a spanContext with the sessionId, traceId, and spanId of the recognized skill invocation.

A skill invocation is recognized through either a SKILL.md filesystem read, which works across frameworks, or a native skill-loading tool in Strands Agents, LangGraph Deep Agents, Google ADK, or the Claude Agent SDK.

Trace placeholders for custom evaluators

AgentCore Evaluations includes two built-in judge-based evaluators for skills. To score something the built-ins don’t cover, create a custom evaluator at the TOOL_CALL level. Tool-level templates can reference skill placeholders:

Placeholder	Contents
{invoked_skill}	Name of the skill loaded on this tool call
{skill_content}	The loaded SKILL.md body
{available_skills}	The catalog offered at runtime, or “(not recorded by this harness)”
{user_message}	The request that triggered the invocation
{context}	The conversation record

For example, a template that checks one specific property of a skill run:

## Skill instructions

{skill_content}

## Conversation record

{context}

## Evaluation Question

Did the agent complete every numbered step in the skill instructions above, in the order given? Answer Yes or No.

The placeholders you reference also decide when the evaluator runs. A template containing {invoked_skill} runs only on skill-invocation spans, and one containing {skill_content} additionally requires the loaded body.

The following commands target the separate skill-enabled runtime. Substitute the runtime name from skills-evaluation/agent_config.json (agent_id or agent_arn) for <skill-runtime>. Strands.SkillInvoked is client-side only and has no CLI equivalent.

Run an on-demand evaluation

Use on-demand evaluation to investigate a session, validate a recent change, or evaluate staged traffic:

agentcore run eval \
  --runtime <skill-runtime> \
  --evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing \
  --session-id <session-id>

Run a batch evaluation

Use batch evaluation to score many stored sessions at once, for example to establish a baseline before a skill-catalog change:

agentcore run batch-evaluation \
  --runtime <skill-runtime> \
  --evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing

Configure continuous online evaluation

Online evaluation samples live traffic so you can detect behavior that a curated test set didn’t anticipate:

agentcore add online-eval \
  --name HRSkillsProductionEval \
  --runtime <skill-runtime> \
  --evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing \
  --sampling-rate 100 \
  --enable-on-create

Then deploy to provision the online evaluation configuration:

agentcore deploy

Continuous evaluation is particularly useful for detecting catalog drift (when a new skill overlaps with an existing description), unanticipated phrasing (when real requests differ from curated test prompts), and long-session failures (when instruction following degrades as context grows).

Best practices

When you’re evaluating agent skills, the first thing to internalize is that routing and execution are two different failure modes, and your evaluation strategy needs to separate them cleanly. If you see a high selection score but a low instruction-following score, that indicates the router picked the correct skill but it was not completely executed. The opposite pattern means the skill would have worked fine if only it had been invoked. Running these two evaluations together, rather than collapsing them into one pass/fail number, is what lets you tell those two stories apart.

Before you build anything custom, start with built-in evaluators to establish a baseline, so any custom logic you add afterward can cover the gap the baseline actually missed. For requirements where you already know the correct routing behavior, don’t rely on a judge model to catch it. Add a deterministic SkillInvoked assertion for every skill that must fire.

After you’re running evaluations, resist the urge to only look at the aggregate score. Per-step evidence is where the real diagnosis happens. It tells you whether an instruction was fully covered, partially completed, or skipped outright, and that level of detail is what turns a failing eval into an actionable fix. This is also why you should evaluate at every lifecycle stage instead of waiting for the final output. A failure at the end doesn’t tell you whether the router sent the request to the wrong place or the right skill executed poorly, and you need both signals to know what to fix.

Skill-level and end-to-end evaluation should be paired, because a skill can execute perfectly and still be the wrong skill for the request in front of it. You will reduce a lot of this ambiguity upstream by writing skill descriptions that are genuinely discriminative. The same logic applies to scope of a skill. A skill built to handle six unrelated functions doesn’t have a single definition of correct behavior, which makes it nearly impossible to evaluate consistently.

State clearly that only the path actually taken in a given run should be evaluated, otherwise untaken branches get miscounted as skipped steps and quietly corrupt your pass rates. And before you trust any of these scores, validate that your extraction pipeline is actually working against live traces.

As you scale this across tools, thresholds may not transfer cleanly. Each evaluation surface needs to be calibrated on its own terms, since a passing score on Strands Evals and a passing score on AgentCore Evaluations aren’t guaranteed to mean the same thing. And ultimately, none of this should live outside your deployment pipeline. Skill quality regressions need to block a release the same way a failing unit test would, or the evaluation work you’ve done up to that point isn’t actually protecting production.

Conclusion

Skills make agents inexpensive to specialize, but a plausible final answer does not prove that the agent selected the right procedure or followed it. Skill Selection Accuracy and Skill Instruction Following separate those failure modes and return evidence for each invoked skill. In Strands Evals, SkillInvoked adds a deterministic guard for known routing requirements. In AgentCore Evaluations, the two judge-based metrics can run on demand for a specific session, as a batch over stored sessions, or continuously over sampled traffic.

Use Strands Evals when you have test cases and recorded trajectories you can rerun. Use AgentCore Evaluations when you want to evaluate OpenTelemetry traces from staged or live agents. Many teams will use both: deterministic and judge-based gates before deployment, followed by trace-based monitoring in production.

Get started

• Strands Evals quickstart.
• Strands Evals repository.
• Amazon Bedrock AgentCore Evaluations developer guide.
• Evaluating AI agents blog post.

Acknowledgements

Thank you to Ritvika Pillai, Vincent Chen, Qiaoxuan Xue, and Shoaib Javed for the AgentCore Evaluations implementation, to Po-Shin Chen for the Strands Evals review, to Anwesan Pal for early discussions on skill evaluation, to Ben Coombs for product guidance, and to everyone else who helped make this work possible.

About the authors

Sangmin Woo

Sangmin is an Applied Scientist at AWS AI Labs, where he conducts research and develops machine learning solutions for agentic AI, with a focus on evaluation frameworks and advancing agent behavior and performance. His interests include agentic AI, generative models, and multimodal AI. Outside of work, he enjoys traveling and exploring new places.

Bharathi Srinivasan

Bharathi is a Generative AI Data Scientist at AWS. She is passionate about Responsible AI to increase the reliability of AI agents in real-world scenarios. Bharathi guides internal teams and AWS customers on their responsible AI journey.

Shruthi Rajoli

Shruthi is a Solutions Architect at AWS based in Chicago, Illinois. She works with startups in the US East region, helping early-stage and growth-stage companies design and build scalable cloud architectures on AWS, with a focus on generative AI, agentic AI workflows, data, and migration workloads. Outside of work, she enjoys walking, yoga, and discovering new food and coffee places.

Visakh Madathil

Visakh is a Solutions Architect at AWS, working with customers and internal teams to bring legibility, trust, and reliability to production artificial intelligence (AI). His work on agentic reliability and AI safety has been presented at machine learning conferences. Outside of work, he enjoys music, birding, and sports.

Renu Rozera

Renu is a Software Development Engineer at Amazon Web Services, where she works on Amazon Bedrock AgentCore. She previously helped build AgentCore Memory and now focuses on developing scalable systems for AgentCore Evaluations & optimization, helping customers assess and continuously improve the quality of their agentic applications.

Vinayak Arannil

Vinayak is a Sr. Applied Scientist at Amazon Web Services. With several years of experience, he has worked on various domains of AI like computer vision, natural language processing, recommendation systems etc. Currently, Vinayak helps build new capabilities on the AgentCore and Strands, enabling customers to evaluate their Agentic applications with ease, accuracy and efficiency.

Haibo Ding

Haibo is a Senior Applied Scientist and Manager working on agentic AI at Amazon. He holds a Ph.D. from the University of Utah. His work focuses on large language models (LLMs) and AI agents, where he leads research in areas such as agent evaluation, agent tool optimization, prompt optimization, and model routing. He has served as an area chair for conferences such as AAAI and ACL, and previously as Program Chair for KDD 2025 Workshop on Prompt Optimization.

Jonathan Buck

Jonathan is a Senior Software Engineer at AWS. He builds agent environments, evaluation frameworks, and post-training infrastructure that help turn advances in agentic AI into reliable production systems.