AI Agent Evaluation: Score Tool Use Before Production

Technology leaders should evaluate an AI agent as an end-to-end business system, not as a chatbot answer generator. A polished final response can hide a wrong tool, invented parameter, skipped approval, unnecessary retry, unverified side effect, unsafe data path, or cost profile that makes the workflow unusable. The production decision therefore needs evidence from the complete task: the requested outcome, every material tool call, resulting system state, final communication, human decision, latency, token use, and total cost.

The release rule is straightforward: no agent goes to production because one answer looks good. Define representative business tasks and expected outcomes, capture the complete trace, score quality and tool behavior at the right level, repeat each scenario, compare the candidate with a fixed baseline in isolation, enforce risk-weighted thresholds, and continue sampling after release. Any low-confidence result or high-impact failure should route to a named person with the trace evidence needed to decide, recover, and learn.

This is distinct from choosing a foundation model. The ITECS guide to blind AI model evaluation tests candidate models against held-out domain data. Agent evaluation tests the assembled production system: model, instructions, skills, retrieval, tools, identities, policies, memory, orchestrator, approvals, user interface, and downstream records. A strong model can still operate inside a weak agent, and a good answer can still come from an unacceptable trajectory.

What AWS clarified on August 26

AWS's August 26, 2026 AgentCore Evaluations post explains how one evaluation service can score agents built with different frameworks. OpenTelemetry and OpenInference traces provide the common evidence layer. AgentCore reconstructs a session from the traces and correlated message events, classifies agent, inference, and tool spans, and then applies the same goal-success, correctness, helpfulness, trajectory, or custom evaluator regardless of the supported framework that produced them.

The date needs precision. AgentCore Evaluations became generally available on March 31, 2026. The August 26 publication explains broader framework-independent operation and its telemetry contract; it is not the service's original launch date. This matters because leaders should evaluate the capability actually documented now instead of treating a new article as proof that every framework, trace shape, or production environment works automatically.

Framework-independent does not mean evidence-independent. AWS documents named support for Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, and the Claude Agent SDK through compatible OpenTelemetry or OpenInference instrumentation. Generic instrumentation must use a recognized scope-name prefix, spans must carry the session identifier needed to reconstruct the run, and the data source must include correlated message content. If a team exports only spans and omits the events holding prompts and responses, response-quality evaluators can fail even though tool-span classification succeeds.

That boundary is useful beyond AWS. Standardized traces reduce the cost of comparing frameworks and moving evaluation logic, but the business still owns trace completeness, semantic consistency, sensitive-data controls, evaluator calibration, retention, and release decisions. An evaluation platform can compute a score. It cannot decide which wrong account update, missed approval, privacy breach, or delayed customer response the organization is willing to accept.

What NVIDIA's August 19 work adds

NVIDIA's August 19, 2026 SkillEvaluator work adds a controlled experimental pattern. Its live tier runs the same task twice in separate sandboxes with the same prompt, model, inputs, harness, and grading criteria—once with a skill and once without it. The difference is reported as Skill Lift. Static validation and security checks remain separate from the live question: did this capability package actually improve the agent's work?

Technology leaders should borrow the design, not overgeneralize the benchmark. A skill is one component of an agent, so positive Skill Lift does not certify the surrounding identity, tools, data, policy, approval, or production workflow. NVIDIA also discloses that most published skill results used only one attempt per task, live runs vary, and the blog's catalog-wide results do not report confidence intervals. Its current documentation describes SkillEvaluator as experimental and community-supported without an enterprise SLA. Those limitations strengthen the case for repeated, company-specific trials before a production decision.

The method still supplies two important controls. First, isolate the candidate so a test cannot alter the baseline's state or inherit its artifacts. Second, change one variable at a time. If a team changes the model, system prompt, tool descriptions, skill, retrieval data, and policy in one release, an improved score cannot identify what helped and a regression cannot identify what broke.

The end-to-end production scorecard

Use the following scorecard as the minimum evidence contract for each agent release. The exact thresholds belong to the business owner, risk owner, engineering owner, and affected operations team. Average scores are not enough: pair them with hard stops for critical safety, authorization, state-change, privacy, and recovery failures.

1. Define business tasks and expected outcomes

Start with recurring decisions or transactions, not generic prompts. For every task class, name the business owner, initiating role, preconditions, approved data, allowed tools, required approvals, expected system state, final response, maximum duration, cost ceiling, and escalation condition. Include an explicit non-goal so the evaluation rewards refusal or handoff when the request falls outside the agent's assignment.

Create representative, edge, adversarial, ambiguous, and negative cases. A procurement agent needs more than an easy request for an approved item. Test an unavailable supplier, conflicting limits, duplicate submission, altered bank detail, missing approver, stale catalog record, prompt injection in an attachment, and a request that should remain manual. Ground truth can be a reference response, a list of behavioral assertions, an expected tool trajectory, a deterministic state check, or a combination.

The outcome must be observable. “Handled the request well” is not a test. “Created no purchase order, returned the policy reason with the correct source, opened an exception for the named owner, and left the supplier record unchanged” can be verified. For tasks with legitimate multiple paths, describe the invariants and acceptable outcomes instead of forcing one brittle transcript.

2. Capture the complete OpenTelemetry trace

Instrument the agent boundary, model calls, retrieval, memory, skill loads, tool calls, approvals, retries, errors, and final response. Preserve session, trace, and span identifiers; agent and release version; model and prompt version; available tools and schemas; tool names, parameters, results, and correlation IDs; timestamps; token counts; and resulting state references. The trace should let an investigator connect the user's request to the action and outcome without reconstructing the run from separate chat screenshots.

Validate telemetry before trusting scores. Run a known test and confirm that the evaluator sees the session, each user turn, inference span, available tool definition, selected tool, parameters, tool result, and final response. Check that buffers flush before short-lived processes exit and that message events arrive in the evaluation data source. Treat missing evidence as a failed evaluation, not a neutral score.

Do not turn observability into a second uncontrolled data store. Filter or tokenize secrets and sensitive payloads, retain protected source data in its system of record, use stable references where possible, restrict trace access, encrypt storage, define retention and deletion, and log access to evidence. Redaction must preserve the fields an evaluator needs; a parameter can be represented by a typed, consistent token without copying the underlying account number into a broad dashboard.

3. Score at the tool, trace, session, and business-state levels

A tool call asks whether the agent selected the right capability and supplied justified parameters. A trace asks whether one request-response turn was correct, helpful, safe, faithful to evidence, and relevant. A session asks whether the whole multi-turn goal was completed and whether the trajectory satisfied its required sequence. A business-state check asks what actually changed in the authoritative downstream system.

AWS's current ground-truth evaluation guidance maps expected responses to correctness, natural-language assertions to goal success, and expected tool lists to exact-order, in-order, or any-order trajectory checks. Use those levels deliberately. A customer-service agent can be helpful at the trace level while failing the session because it never issued the refund. A session can appear complete while a system-state assertion reveals that the refund targeted the wrong order.

Use deterministic checks wherever a machine can establish the fact: schema validity, required field presence, calculated tolerance, authorization decision, record count, identifier match, state transition, or tool-call order. Use calibrated human or LLM judges for semantic dimensions such as helpfulness, tone, and partially structured correctness. Periodically compare automated judges with qualified human reviewers and investigate disagreement; an evaluator is another model or program that can drift or misclassify.

4. Evaluate tool selection, parameters, and call order separately

Tool selection asks whether the chosen action was necessary and appropriate given the request and available alternatives. Measure both commission and omission: selecting a dangerous or irrelevant tool is a failure, but so is answering from memory when the task required an authoritative lookup. Include negative cases in which the correct behavior is to call no tool, ask a question, or escalate.

Parameter accuracy asks whether required values are present, grounded in context, scoped to the intended target, formatted correctly, and free of fabricated additions. High-impact identifiers, amounts, recipients, permissions, dates, and destinations should receive deterministic validation and, where required, independent approval. A correct tool with the wrong customer ID is not a near pass.

Order matters when prerequisites protect the business. The agent may need to authenticate, retrieve current state, verify eligibility, obtain approval, write the change, re-read authoritative state, and notify the requester—in that order. Use exact-order matching when no extra calls are acceptable, in-order matching when diagnostic steps may occur between required actions, and explicit policy tests when approval or cumulative state determines authorization. The ITECS guide to sequence-aware agent authorization explains why individually valid calls can form an unsafe chain.

5. Run repeated tests because agent behavior is nondeterministic

One pass proves only that success can happen. AWS's production guidance notes that the same request can produce different tool selections, reasoning paths, and outputs. Run each important scenario enough times to observe a distribution, not a showcase. Record the model and sampling configuration, tool and service versions, test environment, time window, evaluator version, and every run—not just the best result.

Choose repetition by consequence and variability. Low-impact, deterministic read tasks may stabilize quickly. High-impact writes, open-ended research, long tool chains, rare exceptions, and historically variable scenarios deserve more trials. Report pass rate, failure severity, lower-bound performance, common failure clusters, and worst observed consequence. If the sample is too small for a defensible conclusion, label it preliminary instead of converting a fragile percentage into a production claim.

Repeated tests should include controlled perturbations: reorder harmless source data, vary natural phrasing, change nonmaterial identifiers, simulate latency and tool errors, remove an optional record, and test permission differences. The aim is not to make every run identical. It is to prove that acceptable paths remain inside the same business and safety envelope when the model chooses a different valid route.

6. Compare every candidate with a versioned baseline

Freeze the baseline's agent configuration, prompts, skills, model, tools, policies, evaluation dataset, graders, and environment. Run the baseline and candidate against the same cases, preferably in a randomized paired design, and compare per-case outcomes rather than only two overall averages. A candidate should improve its target dimension without silently degrading safety, latency, cost, or a different business task.

For a prompt or skill change, adopt NVIDIA's with-and-without pattern. For a model change, hold prompts and tools fixed. For a tool-schema change, hold the model and instructions fixed. When an inseparable bundle must change, identify it as a bundle release and do not attribute the result to one component. Store the baseline artifact and score report with the release so the team can reproduce the comparison later.

Avoid shared-state contamination. Give each run its own sandbox or tenant, test identities, records, temporary storage, message queue, clock controls, and cleanup verification. Reset deterministic starting state before every attempt. A candidate that sees the baseline's generated file, cached answer, mutated database row, or approval token has not passed an independent evaluation.

7. Set thresholds that reflect consequence

Define three verdicts for every candidate: quality, control, and operations. Quality covers business goal completion, correctness, and helpfulness. Control covers authorization, privacy, safety, tool behavior, approval, audit, and recovery. Operations covers availability, latency, tokens, tool calls, review load, and cost per accepted task. Production requires all three verdicts to pass; a high helpfulness score cannot compensate for an unauthorized write.

Use both floors and hard stops. A team might require a minimum repeated goal-completion rate and a noninferior response-correctness score, while allowing zero critical authorization failures, zero uncontained high-impact writes, and zero unrecoverable state changes. Set maximum p95 latency, retry count, token use, review rate, and cost per completed task. Define who may approve an exception, how long it lasts, and what evidence must accompany it.

Keep a holdout set that prompt and skill authors do not tune against. Add every material incident, near miss, user complaint, evaluator disagreement, and newly discovered edge case to a protected regression set after investigation. Rotate or expand cases when source data, users, tools, policies, or threat patterns change so the suite measures the current job instead of a memorized historical sample.

8. Sample live traffic after production

Preproduction evaluation cannot reproduce every user, data state, service delay, permission combination, or adversarial input. AWS's current evaluation modes support targeted on-demand work, batch comparisons, and online sampling with filters and score trends. Use sampled live evaluation to detect drift, recurring low scores, changed tool behavior, and task categories the original dataset missed.

Design sampling by risk, not convenience. Random sampling estimates overall quality; targeted sampling should cover high-impact tools, new releases, rare workflows, low-confidence outputs, policy denials, retries, long sessions, unusual costs, complaints, and new user groups. Preserve privacy boundaries and exclude or transform data that the evaluation system is not approved to process.

Online evaluation has a ground-truth limitation: production traffic normally does not arrive with a known expected response or trajectory, and AWS does not allow custom online evaluators that depend on ground-truth placeholders. Pair live semantic scoring with deterministic business-state checks, policy events, sampled human review, customer feedback, and later adjudication. A live score is a signal for investigation, not proof that the action was right.

9. Route uncertainty and consequence to human review

Do not treat the agent's self-reported confidence as the gate. Route when evaluator confidence is low, evaluators disagree, required evidence is missing, the request is ambiguous, a tool result conflicts with authoritative state, a high-impact threshold is reached, policy denies the action, or the run falls outside the evaluated task envelope. The reviewer should receive the relevant trace, expected outcome, actual tool path, state diff, evaluator explanations, and recommended recovery—not a blank approval button.

Name the review owner and response target for each task class. Separate people who can evaluate quality from people who can authorize the action when duties require it. Record approve, reject, correct, rerun, escalate, contain, and roll-back decisions with reasons. Feed adjudicated failures back into the evaluation set, but do not let an agent automatically rewrite its own release threshold or suppress the evidence that caused an escalation.

10. Measure cost per completed, accepted task

Latency, token use, and unit price are inputs; completed accepted work is the denominator. Calculate total model, tool, retrieval, runtime, observability, evaluator, human-review, retry, and correction cost for tasks that meet the business acceptance criteria. Track failed and abandoned runs separately. A cheaper model can cost more if it loops, calls unnecessary tools, generates excessive context, or increases manual correction.

AWS's AgentCore Observability guidance exposes operational signals including latency, duration, token use, sessions, and errors through OpenTelemetry-compatible data. NVIDIA similarly reports token use and execution time separately from its efficiency score because a capability may improve correctness while consuming more resources. Preserve that separation: quality, time, tokens, and money answer different questions.

Set service targets by workflow. A cyber-response agent may justify premium latency and cost if seconds change containment; a nightly reconciliation may favor batch economics; a customer-facing workflow may optimize time to a verified answer; a research agent may accept longer duration but cap tool spend and human review. Compare candidates on the same workload and report cost alongside the quality and control verdicts.

11. Preserve trace evidence for debugging and audit

Version every evidence package with the agent release, test-set version, evaluator and rubric versions, model, prompt, skills, tool schemas, policies, identities, environment, run IDs, scores, explanations, deterministic assertions, human adjudications, timestamps, resource use, and final release decision. Link production incidents back to the exact deployment and baseline. Protect the package from routine agent modification.

Retain enough evidence to answer what the user requested, what the agent saw, which tools were available, what it selected, which parameters and results moved through the chain, which approvals and policies applied, what system state changed, what it told the user, why the evaluators scored it, and how the team responded. Define retention with security, privacy, legal, compliance, and records owners; “keep everything forever” is not an audit strategy.

A useful evaluation program ends with an operating loop: observe, detect, investigate, adjudicate, add the failure to the regression set, change one bounded component, compare against the baseline, pass the full gate, release, and monitor again. Connect that loop to AI DevOps so prompt, model, skill, tool, policy, and evaluator changes receive the same versioning, evidence, rollback, and ownership discipline as other production software.

A practical 30-day evaluation rollout

During week one, select one bounded agent workflow and write its task contract, negative cases, authoritative state checks, owners, and consequences. Instrument the complete trace and validate that sensitive data handling and session reconstruction work. During week two, calibrate tool-, trace-, session-, and state-level evaluators with qualified reviewers, then set preliminary quality, control, and operations thresholds.

During week three, freeze the baseline and run repeated, isolated paired tests. Analyze per-case regressions, worst failures, evaluator disagreements, latency, tokens, tool calls, review time, and cost per accepted task. Fix the system rather than lowering a threshold to meet a date. During week four, test rollback and human escalation, approve the release evidence package, start with a limited production audience, and enable risk-based live sampling.

Reopen the decision whenever the model, prompt, skill, tool description or schema, retrieval source, policy, permission, memory, evaluator, tracing library, framework, runtime, or business acceptance criteria changes. Also re-run after an incident, a material drift signal, an unexplained cost increase, or an update to AgentCore or SkillEvaluator capabilities. Production readiness is a versioned finding, not a permanent badge.

The operating principle: score the path, not just the prose

A final answer is one artifact at the end of an agent's work. The business risk and value live in the path that produced it. Evaluate whether the right goal was completed, the right tools were chosen, the right values were sent in the right order, the side effects match authoritative state, critical controls held, behavior remains dependable across repeated runs, and the accepted result arrived within the time and cost the business approved.

AWS makes that evidence more portable across instrumented agent frameworks. NVIDIA shows how to isolate a capability and compare it against a baseline. Neither replaces the organization's task definition, release threshold, human judgment, or accountability. Use the platforms as measurement infrastructure; keep the production decision with the people who own the workflow and its consequences.