An AI agent can return a polished answer, finish quickly and produce no application error while giving the business the wrong information. AI agent regression testing addresses that gap: compare each change with trusted examples, then keep checking real outcomes after deployment. Uptime and fluent writing are not evidence of correctness.
Start with one business task, version its expected outcomes and test the tools, individual agent and complete workflow separately. Combine deterministic checks with reviewed scoring rubrics, block unacceptable releases, and watch live quality against the same business contract. When a confirmed failure occurs, contain it, recover the affected workflow and turn the case into a permanent test.
An AWS engineering post published October 16, 2026 describes an AgentCore-based evaluation approach spanning release checks and production monitoring. It highlights an important distinction: quality can deteriorate when upstream data changes, even without an application deployment. The framework below is our proposed operating plan, not an AWS performance claim or a guarantee that evaluation catches every error.
Define quality in business terms before choosing a grader
Write a task contract with the process owner. Name the decision being supported, permitted sources and tools, required facts, effective dates, units, allowed actions and circumstances that require escalation. A support answer may need the applicable policy version and an explicit admission that a customer's eligibility cannot yet be determined. A finance answer may need an as-of date and a distinction between booked revenue and a forecast.
Separate correctness from usefulness, safety, speed and cost. A concise answer can be incorrect; a correct answer can disclose restricted information; a safe refusal can still fail an ordinary allowed task. Keep these dimensions visible rather than combining them into one score that lets good prose offset a critical defect. Our agent-evaluation guide covers the broader system assessment; this guide focuses on detecting deterioration over time.
Keep golden and sentinel sets at three levels
Use a golden set as the broader, versioned collection of representative tasks with approved expectations. Use a sentinel set as a smaller, risk-focused suite that runs frequently and detects known failure patterns quickly. Maintain explicit membership, owners and versions. If sentinels are a subset of the golden set, do not present their results as independent evidence.
The following is a proposed test inventory. Give each level its own results so an end-to-end failure can be traced to a broken tool, an agent decision or a handoff. Include normal work, boundary cases, unavailable dependencies and legitimate refusals. Keep development examples separate from a protected holdout; tuning repeatedly against a supposedly final test weakens its value.
On a narrow screen, scroll horizontally to compare the test sets.
| Level | Golden coverage | Frequent sentinels |
|---|---|---|
| Tool | Typed inputs, authoritative outputs, denied access, empty results and timeouts. | A few critical fact, permission and error-contract cases. |
| Agent | Task outcomes, evidence, allowed tool choices and escalation across representative cases. | Known failure triggers: wrong units, stale dates, unsupported claims and unsafe actions. |
| Multi-agent workflow | Delegation, evidence handoffs, conflicting findings and the final business outcome. | Dropped constraints, changed identity, lost citations and unresolved disagreement. |
For multi-agent tests, verify what crossed each boundary: task scope, identity, evidence, unresolved uncertainty and remaining authority. Two agents repeating the same unsupported assertion do not corroborate it. The final agent must reconcile conflicting evidence or escalate, not select whichever answer sounds more confident. Test these handoffs with fresh state and bounded turns.
Use exact checks for facts and explicit rubrics for judgment
When the expected result is known, check it directly. Validate identifiers, required fields, arithmetic, dates, currencies, units, permissions and allowed side effects against the approved reference. Define tolerances and normalization before scoring. Do not normalize away a meaningful difference, such as dollars versus thousands of dollars. Missing evidence or a failed checker produces an unknown or failed evaluation, never an automatic pass.
For explanations and recommendations, score observable qualities: whether the answer addresses the task, uses the supplied evidence, states important constraints and handles uncertainty appropriately. A proposed rubric might give each criterion 0 for absent or contradicted, 1 for incomplete, and 2 for fully supported. Attach concrete examples and adjudicate disagreements. Evaluate the answer and recorded actions, not a demand for private chain-of-thought.
Anthropic's agent-evaluation guidance recommends combining grading methods and warns that rigid tool-sequence expectations can reject valid solutions. Calibrate model judges against human decisions, repeat trials and isolate runs. Require a particular call order only when the task or safety contract makes that order necessary.
Treat evaluator prompts, model versions and rubrics as versioned dependencies. Test the judge with correct, incorrect and adversarial answers. Untrusted text in a transcript must not instruct the evaluator to change its score or execute a tool. Review grading failures separately from agent failures, and retain the reason for each adjudicated label.
Freeze tool responses to isolate regressions
For a replay test, save approved tool requests and responses with their schema, permission context, error cases and relevant timestamps. Replace external calls with those fixtures in an isolated harness. An unmatched call should fail visibly, not silently reach a live service. Reset memory and state between trials, and prohibit real writes.
Replay answers a narrow question: did the candidate behave differently when given the same tool evidence? It does not prove that today's API, credentials, network or source data still work. Test the tool contract separately against a controlled integration environment, then use live checks for current evidence. Measure real latency in that environment rather than treating replay timings as production performance.
Run baseline and candidate on the same cases, fixture versions and evaluation rules. Repeat stochastic cases with a predetermined run count; do not rerun only failures until they pass. Report paired differences, cohort sizes and uncertainty. A small score change on a tiny sample is an investigation prompt, not a confident improvement claim.
Make release evidence travel with the agent
Bundle code, prompts, model identifiers, tool schemas, retrieval configuration, permission policy and evaluator versions into a traceable release record. Attach dataset versions, results, unresolved findings and the approver. Record which versions actually executed, especially when a provider uses a changeable model alias.
Run the fast sentinel suite on relevant code, prompt, tool and configuration changes. Before promotion, run the broader golden set and protected holdout, including critical negative cases. Require agreed accuracy floors, no observed critical safety violations, acceptable latency and cost, and a supported recovery target. A failed grader or missing test cohort should not disappear from the release receipt.
Set the allowable deterioration before evaluation. For example, the owner might permit a small change in low-risk phrasing scores but no failure of designated critical facts or authorization checks. This is a risk decision, not a universal percentage. Use agent boundary tests alongside quality checks so a useful answer cannot compensate for an unauthorized action.
Score canaries and watch live drift windows
Begin with a narrow, reversible rollout. Compare the candidate and accepted version on comparable tasks, accounting for differences in traffic and evidence freshness. Synthetic canary questions can exercise known live facts; a limited production sample can reveal cases the test author did not anticipate. A shadow run must not duplicate customer messages, payments or other writes.
LangSmith's evaluation concepts distinguish offline datasets with expected outputs from online evaluation of production runs. That distinction matters operationally: a live rubric score is not automatically a factual truth label. Attach an authorized reference where possible; otherwise report a quality signal awaiting validation.
Define the window duration, minimum scored count, cohort and escalation rule before enabling alarms. For ordinary degradation, require enough observations to distinguish noise from a meaningful change. Critical unsafe behavior may require immediate containment instead of waiting for a window to fill. Sparse windows remain insufficient evidence, not green results. Account for scoring queues and delayed human labels when setting detection objectives.
Sample across task type, language, tool path and risk, while retaining a random sample of apparently successful work. Review failures, abstentions and expensive retries as well. Report results by cohort before aggregating them; a shift toward easier questions can conceal deterioration on important work. Distinguish changes in task mix, source freshness and evaluator behavior from an actual agent regression.
Example: grounded in the wrong policy
Consider a hypothetical support agent that answers return-policy questions. Its approved reference says a specific product category has a 14-day return window. A retrieval index accidentally points to an archived policy that says 30 days. The agent quotes the archived document accurately and returns a fluent answer. Grounding in the retrieved text passes, but factual accuracy against the current applicable policy fails.
A frozen replay using the previously approved evidence still passes. A live canary that checks the current policy version detects the mismatch. The operator pauses this answer category, routes requests to a human and verifies the tool's retrieved document identifier and effective date. If the defect is the index selector, restoring an older model alone will not fix it.
After restoring the approved index configuration, the team checks fresh answers, identifies affected responses for review and records recovery evidence. It adds the archived-versus-current policy conflict to tool tests, the missing effective-date warning to agent tests, and the dropped policy version to a handoff test. These are illustrative controls, not a reported ITECS client incident.
Choose acceptance measures with honest denominators
Use the proposed scorecard below to agree numerical floors, ceilings and response objectives with the business owner before the pilot. Thresholds depend on consequences and available evidence. Keep critical cohorts separate and publish sample sizes; zero observed errors in a small test does not establish safety for all future work.
On a narrow screen, scroll horizontally to read definitions and acceptance considerations.
| Measure | Definition | Acceptance consideration |
|---|---|---|
| Grounding | Supported material claims / material claims checked, using the cited evidence. | Set a floor per risk class; missing evidence is not a pass. Check source authority and freshness separately. |
| Factual accuracy | Correct required facts / required facts in the adjudicated task set; omissions fail. | Use dated authoritative answers and fixed normalization. Require all designated critical facts to pass. |
| Tool selection | Runs using permitted, task-appropriate tools and valid arguments / evaluated runs. | Test necessary ordering constraints, not one arbitrary path. Track unnecessary calls and missing required calls. |
| Safety | Unauthorized attempts, actual effects and correctly stopped unsafe requests, each reported separately. | Block release on an observed critical violation. Zero observed violations is not proof of zero risk. |
| Latency | p50 and p95 request-to-accepted-outcome time, with timeouts and failures reported. | Meet the business time budget; do not hide slow failures by measuring only successful model calls. |
| Cost per task | Model, tool, retry, infrastructure and review cost / correctly completed tasks. | Set a ceiling; show failed-task spend and evaluation overhead separately. An empty denominator is undefined. |
| False alarms | Adjudicated non-regressions / adjudicated regression alerts in the window. | Set an acceptable alert burden and report unresolved alerts. Low alert volume alone does not establish detection quality. |
| Regression escape rate | Confirmed regressions first discovered in production / confirmed regressions found before or after release in the tracked release cohort. | Set a tolerated rate by severity and retain a follow-up cutoff. Also report affected production tasks and later discoveries. |
| Detection time | Time from earliest evidenced impact to an actionable quality alert. | Set a severity-specific limit; report an interval when the start is uncertain. Do not substitute the alert timestamp for impact. |
| Recovery time | Time from actionable alert to verified restoration of acceptable outcomes. | Meet the recovery objective in an exercise. A rollback command succeeding is not proof of restored quality. |
| Review and coverage | Review minutes per adjudicated case; evaluated tasks / eligible tasks, by cohort. | Set review-capacity and minimum-sample requirements. Label insufficient evidence rather than treating unscored work as good. |
A useful acceptance record names the owner, threshold, sample requirement, evidence location and stop action for every measure. Do not label unknown factual accuracy as perfect grounding, a blocked unauthorized attempt as harmless intent, or an unevaluated task as a success. Reconcile all submitted evaluation cases to passed, failed, inconclusive or infrastructure-error states.
Connect alarms to containment and tested recovery
Route each alarm to an accountable operator with the affected release, task cohort, evidence link and severity. Provide a way to pause consequential actions independently of changing the model. Preserve the accepted release bundle and test restoration of its compatible tools, retrieval configuration and permissions. Keep a manual path if a provider retires the old model or a data dependency cannot safely be reversed.
Rollback does not undo messages already sent or records already changed. Identify in-flight and completed work from the affected version, prevent duplicate execution during recovery and assign review or correction owners. Verify restored business outcomes before closing the incident. Record both detection and recovery time, including uncertainty about when the problem began.
Protect evaluation evidence like production data. Minimize or de-identify fixtures, restrict transcript access, keep secrets out of replay files and agree retention for traces, labels and adjudications. Use opaque case identifiers and controlled evidence links in alerts rather than copying sensitive customer content into chat channels.
Turn real failures into permanent tests
Have a domain reviewer confirm each feedback case before promoting it into the suite. Preserve the original evidence under appropriate access controls, then create a minimized reproducible fixture with the correct outcome and failure rationale. Add a neighboring legitimate case so the fix does not simply refuse everything similar. Assign an owner and review date when the expected fact can expire.
Version the updated dataset and rerun both the old cases and the new ones. Keep historical results comparable rather than overwriting yesterday's labels without a record. Reserve fresh holdout examples for final acceptance, and periodically audit false alarms so a noisy grader does not train operators to ignore the real failures.
For a staged pilot, first establish an offline baseline for one task; next rehearse a deliberately introduced defect and recovery in isolation; then score a limited live cohort without expanding action authority. Broaden use only when the business owner accepts quality, review burden, detection and recovery evidence. An evaluation dashboard is useful only if someone can act on what it finds.
ITECS can help your team structure AI DevOps around repeatable testing, controlled releases and operational ownership. Talk with ITECS about a bounded regression-testing pilot for your custom AI agents, with clear evidence for when to ship, stop and recover.
