Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore

來源原標題

Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore

作者

Sangmin Woo

原始標籤

Advanced (300) · Amazon Bedrock AgentCore · Announcements

原始連結

https://aws.amazon.com/blogs/machine-learning/evaluate-skill-equipped-agents-with-strands-evals-and-amazon-bedrock-agentcore/

摘要原文

General-purpose agents handle a broad range of tasks, but you still need them to follow the procedures that run your business: compliance checks, document-processing workflows, escalation policies, engineering conventions. Encoding all of that in one system prompt or in application logic gets hard to maintain and update. Skills are a modular alternative. A skill is a reusable set of instructions, usually stored in a SKILL.md file, that teaches an agent a domain-specific task like redacting a contract, reconciling an invoice, or following a team’s pull-request conventions. Because skills follow the open Agent Skills standard, they are portable across compatible harnesses, and the agent loads only the skill it needs at runtime instead of carrying every procedure in its core instructions. A skill packages one or more tools with the context an agent needs to use them correctly: This modular approach helps teams specialize agents faster, reuse proven procedures across agents and workflows, keep behavior consistent, and update domain-specific guidance without fine-tuning the underlying model or rewriting the agent’s core logic. This skill composability in agents introduces two failure modes that general output-quality metrics can miss: the agent invokes a skill that is not appropriate for the task and the agent invokes the right skill but skips or only partially follows its instructions. Both failures can produce a fluent, plausible response without having used your pre-determined domain knowledge. An evaluation therefore cannot examine the final response alone. To make these failures measurable, Strands Evals SDK and Amazon Bedrock AgentCore Evaluations , a capability of Amazon Bedrock AgentCore, add skill-focused evaluators: In this post, you will learn how to evaluate skill selection and instruction following from a recorded trajectory in Strands Evals, add deterministic routing checks to a test suite, evaluate skill behavior from OpenTelemetry traces with AgentCore Evaluations, and interpret per-skill results to choose the right fix, all through the AgentCore CLI. An agent receives a task and a catalog of skills, chooses a skill, loads it, and acts. The run is recorded as a trajectory in Strands Evals or an OpenTelemetry trace in your observability layer. This record can now be used for all three skill evaluators. Skill Selection Accuracy checks whether each invoked skill fits the task and whether the agent invoked the correct skill. The following figure shows how the agent chooses a skill from the 1:n skills provided to it. Skill Selection Accuracy then scores whether the selected skill is the correct one for the task. You can find the prompt template and the rubric of this evaluator in the prompt template documentation . Figure 1: Skill Selection Accuracy checks whether the agent chose an appropriate skill for the task Skill Instruction Following asks how fully the agent followed that skill’s prescribed steps. You can find the prompt template and the rubric for this evaluator in the prompt template documentation . Figure 2: Skill Instruction Following measures how completely the agent followed the skill’s steps SkillInvoked is deterministic. It calls no model and is specific to Strands Evals. An overview of these skill evaluators is demonstrated diagrammatically in the following figure. Figure 3: Overview of the three skill evaluators Consider an HR assistant agent with skills for paid time off (PTO) planning and discussing employee benefits. An employee asks about their dental and vision benefits. If the agent invokes the benefits skill, it may produce a more polished response. If the tool call succeeded but the agent chose the wrong playbook, Skill Selection Accuracy isolates that routing decision. Now suppose the agent correctly invokes the PTO-planning skill for a related request. The skill instructs the agent to identify the employee_id , check the PTO balance, check the rollover rules against the latest HR policy, and then submit a PTO request if the conditions allow. If the agent checks the PTO balance but skips the rollover rules, the agent might still return a plausible response while violating the prescribed process. Skill Instruction Following isolates that execution failure and identifies the skipped step. The failures require different fixes. An inappropriate selection often points to overlapping or ambiguous skill descriptions. Incomplete instruction following might call for clearer steps, a different skill structure, or a more capable agent model. Because judge-based evaluators return per-invoked-skill results, multi-skill runs remain diagnosable: you can identify which selection or instruction-following result lowered the aggregate score. If no skill is invoked, the judge-based evaluators don’t produce a score. Pair them with SkillInvoked when a regression test has a known routing requirement. To follow the Strands Evals section, install the SDKs: You also need a recorded agent run. The skill evaluators accept either a Strands Evals Session or a raw message list as the trajectory. At launch, skill extraction recognizes signals from the Strands AgentSkills plugin, Claude Code, Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK, and generic SKILL.md file reads. To follow the AgentCore Evaluations section, you need: The examples in this post use the AgentCore CLI: Strands Evals is useful when you control the test cases and can rerun the agent during development or continuous integration. Check out the complete Strands evals code sample created for the HR assistant agent in the complete code sample . The skill evaluators read the run’s trajectory. TracedHandler collects the agent’s spans and attaches them as the trajectory: Suppose the agent loaded pdf-table-extraction , ran pdftotext -layout , but never opened the extracted file or located the table boundaries. A simplified report: Skill Instruction Following uses five ratings: Fully Followed (1.0), Mostly Followed (0.75), Partially Followed (0.5), Minimally Followed (0.25), and Not Followed (0.0). It passes at Mostly Followed or better. AgentCore Evaluations works directly with existing OpenTelemetry traces, supporting on-demand evaluation, batch processing of stored sessions, and continuous sampling of live traffic. Telemetry is organized into sessions, traces, and spans. Because skill evaluators operate at the tool-call level, each result includes a spanContext with the sessionId , traceId , and spanId of the recognized skill invocation. A skill invocation is recognized through either a SKILL.md filesystem read, which works across frameworks, or a native skill-loading tool in Strands Agents, LangGraph Deep Agents, Google ADK, or the Claude Agent SDK. AgentCore Evaluations includes two built-in judge-based evaluators for skills. To score something the built-ins don’t cover, create a custom evaluator at the TOOL_CALL level. Tool-level templates can reference skill placeholders: For example, a template that checks one specific property of a skill run: The placeholders you reference also decide when the evaluator runs. A template containing {invoked_skill} runs only on skill-invocation spans, and one containing {skill_content} additionally requires the loaded body. The following commands target the separate skill-enabled runtime. Substitute the runtime name from skills-evaluation/ agent_config.json ( agent_id or agent_arn ) for <skill-runtime> . Strands.SkillInvoked is client-side only and has no CLI equivalent.