An LLM evaluation step in CI needs three classes of check, because model output varies from run to run. Contract checks block a merge on any failure. Quality scores block only on a sustained drop below the baseline, main’s score on the same cases. New metrics run in observe mode and fail nothing until their judge, the model that scores the output, agrees with your experts on a labeled set of examples.
Four pieces of pipeline work make those checks trustworthy. The eval runs whenever a prompt, a model setting or a retrieval setting changes. A manifest, a file in the repository, pins every input to the score. Judge calls have ceilings. One required check carries the verdict, and deploys wait on a full run of every case.
The examples use GitHub Actions. Names, thresholds, tolerances, sample sizes and costs in them are illustrative and labeled as such.
What a prompt change can break that unit tests cannot see
A unit test feeds code fixed inputs and expects one answer. Model output does not hold still that way. Anthropic’s glossary says that even with temperature set to 0, “identical inputs might produce different outputs across API calls.”
So a one-line prompt edit can pass every unit test and still ship a regression. The output stops matching the JSON schema the next service parses. The model picks a different tool or drops an argument. It answers a question it used to refuse. Its answers drift away from the retrieved documents. It writes longer answers, and the token bill grows with them. Each of these reaches production as a successful API response.
DORA’s 2025 report, announced September 23, 2025 and drawn from nearly 5,000 technology professionals, found AI adoption still has a negative relationship with software delivery stability. It names strong automated testing among the control systems that keep added change volume from turning into instability. The report studies AI used to build software. In a product that calls a model, the eval is the automated test that can see a prompt change.
Run the eval when prompts, model settings or retrieval change
List every file that can change what the model sees or how it answers. That covers prompt templates, the model ID and decoding settings, and retrieval settings such as the index version, top-k (how many chunks a search returns) and chunking (how documents are split). It covers the tool definitions the model can call. It also covers the eval’s own dataset, judge prompt and rubric, the written criteria the judge scores against. Prompts assembled in application code count too, so list those source paths or move the text into template files under version control.
Keep that list out of the workflow’s trigger. GitHub’s guide to troubleshooting required status checks says a workflow skipped by path filtering leaves its checks “Pending,” and they block merging. A pull request that changes nothing on the list then waits for a status that never arrives.
Run the workflow on every pull request and decide inside it. A first job compares the diff with the list. The eval job runs only when that job reports a match, and GitHub reports a job skipped by a conditional as a success. A last job carries the verdict. It runs with always(), because a job that depends on a failed job is skipped, and GitHub warns that skip may not block merging.
on:
pull_request:
merge_group:
concurrency:
group: evals-${{ github.ref }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-latest
outputs:
ai: ${{ steps.diff.outputs.ai }}
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- id: diff
run: ci/changed.sh >> "$GITHUB_OUTPUT"
evals:
needs: changes
if: needs.changes.outputs.ai == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v6
- run: ci/run-evals.sh evals/manifest.yaml
eval-gate:
needs: [changes, evals]
if: always()
runs-on: ubuntu-latest
steps:
- run: |
test "$CHANGES" = success
test "$EVALS" != failure
test "$EVALS" != cancelled
env:
CHANGES: ${{ needs.changes.result }}
EVALS: ${{ needs.evals.result }}
The gate fails unless the diff job succeeded and the eval job passed or was skipped. A cancelled or timed-out eval counts as a failure.
Pin a manifest so two runs can be compared
A score can be compared with another only when both runs tested the same thing. The manifest records that, and the runner stores the manifest’s hash beside every score. The eval owner, a named engineer who keeps the suite, maintains the file. Changes to it go through the same review as code.
Pin the model to a snapshot, a model version that stays fixed behind its ID. Anthropic’s model ID guide says the model behind an ID “remains constant for the lifetime of that ID.” Older Claude models also have aliases such as claude-sonnet-4-5, which point to the most recent dated snapshot. An alias in the manifest lets the model change without a commit.
Azure OpenAI sets this per deployment. Microsoft’s model versions page lists three upgrade policies for Standard deployments, and two of them move a deployment to a new version on their own. Opt the deployments your gate pins out of automatic upgrades. An opted-out deployment stops working when its version retires, so the retirement date goes on the eval owner’s calendar.
Prompt registries need the same care. MLflow’s Prompt Registry makes prompt versions immutable, while its aliases are mutable pointers. The model configuration attached to a prompt version can also change after the version is created. Pin the version number, and copy the decoding settings into the manifest instead of reading them from the registry.
- Prompt
support-answer, version 14A registry alias can move to another version. Pin the version number.- Model snapshot
claude-haiku-5-5An ID that maps to one snapshot. An alias or an auto-upgrading deployment changes the model without a commit.- Decoding settings
temperature 0.2, max_tokens 800Copy the values. A registry’s attached model configuration can change after the prompt version is created.- Retrieval settings
index build 2026-10-20, top-k 6, chunks of 800 tokensA rebuilt index or a new top-k changes what the model reads.- Dataset
support-evals v9, with its content hashAdded or edited cases move the score with no product change.- Rubric and judge
rubric v4, judge claude-sonnet-5-5A new rubric or judge snapshot needs recalibration and a new baseline.- Gate settings
3 runs, tolerance 0.04, ceilings of 400 calls per pull request and 3,000 per full runThe tolerance is how far below the baseline a score may fall. Each ceiling caps the judge calls of one kind of run.- Baseline
ten nightly runs on main, 2026-10-24 to 2026-11-02Their stored per-case scores. The baseline moves only by a reviewed commit, so it never drifts down on its own.
The hash covers the six fields that change what a score means. Gate settings and the baseline sit outside it, so a new tolerance or a raised ceiling keeps the baseline. A pull request that edits a prompt is compared with main under the same dataset, rubric and judge. A pull request that changes one of those changes the measure itself. That run re-scores main under the new manifest first and blocks on contract checks only. Keep measure changes and product changes in separate pull requests, so a score change has one cause.
Three gate classes: block, compare, observe
Each check belongs to one class, which decides what a failure does to the pull request.
- Block
- Fails the build onAny single failure.ChecksContract checks graded by code. The output parses against its schema, tool calls name a real tool with valid arguments, must-refuse cases set the schema’s refusal field, and cited sources come from the retrieved set.OwnerThe eval owner maintains the checks. The pull request’s author fixes the change.
- Compare
- Fails the build onA sustained drop. The median of the repeated runs falls below the baseline minus the tolerance.ChecksQuality scores against the baseline, such as faithfulness to the retrieved documents or a calibrated judge’s rubric score.OwnerThe product owner sets each tolerance with the eval owner.
- Observe
- Fails the build onNever. Scores go to the run summary and the results store.ChecksNew metrics, and any metric whose judge is still in calibration.OwnerThe eval owner promotes a metric to compare once its judge passes calibration.
Contract checks run as code against a fixed rule. Anthropic’s guide to building evaluations ranks code-based grading as the fastest and most reliable method. It says LLM-based grading needs testing “to ensure reliability first, then scale.”
A contract check that fails one run in twenty is still a defect, because one in twenty requests like that case would get that output. Fix the cause in code, for example with a schema-constrained output or a validated retry, and keep the check itself strict.
Contract checks run on the sentinel set, the small set of high-risk cases that runs on every change, plus the cases tagged to whatever changed. AI agent regression testing covers how to build and maintain those sets. For agents, the tool-call checks belong in this class, and scoring tool use before production covers how to score them.
Block on a sustained drop below the baseline
A quality score varies between runs of unchanged code, so a gate on one run fails good changes at random. Measure that noise first. Store every case’s output and score from each nightly run on main, all under one manifest.
A pull request runs only its own cases, and a score over 50 cases swings wider than one over 400. So take the baseline and the spread from the stored nightly scores on the pull request’s own cases. The gate uses the tolerance or that spread, whichever is wider.
The product owner for the feature sets each tolerance with the eval owner, because it states how much quality loss the business accepts. The baseline names the nightly runs it comes from. Both sit outside the manifest’s hash and move only by a reviewed commit, so small drops cannot lower the bar one merge at a time.
Then run the candidate a fixed number of times and gate on the median. Ragas’s faithfulness metric scores answers built on retrieved documents. It divides the claims the retrieved context supports by all the claims in the response, so it ranges from 0 to 1. Ragas uses an LLM to break the response into claims, and an LLM or a small classifier model to check each one. That makes faithfulness a judged metric, and it needs calibration like any other.
- Baseline
- 0.86
- Main’s median on this pull request’s 50 cases, from the stored scores of the ten pinned nightly runs.
- Tolerance
- 0.04
- Set by the product owner. On these 50 cases the ten runs ranged from 0.84 to 0.88, no wider than the tolerance. The floor is 0.82.
- Candidate A
- Median 0.84Passes
- Three runs scored 0.85, 0.81 and 0.84. One run fell below the floor and the median held, so it passes.
- Candidate B
- Median 0.79Blocks
- Three runs scored 0.80, 0.79 and 0.78. The median sits below the floor, so it blocks the merge.
Candidate A passes although one run fell below the floor. Candidate B blocks because its median fell below the floor. Decide the run count before the eval starts. Rerunning a failed candidate until it passes turns the gate back into a coin toss.
Calibrate the judge before it can fail a build
A judge that disagrees with your experts gates on the wrong thing. In Ragas’s judge alignment guide, the first judge agreed with expert pass and fail labels on 121 of 160 responses. All 39 disagreements were lenient, meaning the judge passed answers the experts had failed. A rewritten judge prompt raised agreement to 139 of 160. Even that falls short of the illustrative bar of 9 in 10 in step 4 below. Whether a judge that good may gate is the product owner’s call.
A lenient judge lets regressions merge, and a strict one blocks good changes. Measure both directions before the judge can fail a build.
Have a domain expert label the calibration set.
The calibration set is a sample of real outputs with expert pass or fail labels. Ragas calls 100 to 200 representative examples sufficient and recommends a principal domain expert. The product owner names that expert, and the prompt’s author does not label.
Include outputs the expert fails.
A set of good answers cannot show a lenient judge. Keep failing outputs from past incidents and from deliberately broken prompts in the set.
Score the stored outputs with the judge.
Ragas scores stored responses instead of generating new ones, so the comparison measures the judge alone. Record the judge’s model ID and the rubric version with the result.
Count disagreement in each direction.
A lenient miss is a judge pass on an expert fail, and a strict miss is the reverse. The product owner sets a bar for each. Illustrative bar: agreement on at least 9 of every 10 examples, and no lenient miss on a must-fail case.
Fix the rubric, then promote the metric.
Anthropic’s eval guide advises detailed rubrics, a fixed output such as correct or incorrect, and a grader with thinking on, which the budget below prices. Re-score the whole set after each change. The eval owner moves the metric to the compare class once it clears both bars.
Recalibrate whenever the judge’s snapshot or the rubric changes. Both are manifest fields, so the pull request that changes them is the reminder.
Cap judge calls per run, and run only the affected cases
Judge cost is the number of cases, times the repeat runs, times the judge calls per case. A pull request needs the cases its change can affect. The nightly run on main needs every case, and its stored scores supply the baseline, so a pull request pays for its own runs only.
Send the nightly run through a batch API where the provider offers one. Anthropic’s Message Batches API cuts token prices in half, and most batches finish within an hour. Batches still unfinished after 24 hours expire. That suits a nightly run and rules batch out for a pull request or a deploy someone is waiting on.
- Pull request
- $4.20per run
- The sentinel set plus the cases tagged to the change: 50 cases, 3 runs, 2 judge calls each.Judge calls300Ceiling400List priceStandard: $2 input and $10 output per million tokens.
- Nightly run on main
- $16.80per run
- All 400 cases, 3 runs, 2 judge calls each.Judge calls2,400Ceiling3,000List priceBatch: $1 input and $5 output per million tokens.
- Deploy
- $33.60per run
- All 400 cases on the commit being deployed, 3 runs, 2 judge calls each.Judge calls2,400Ceiling3,000List priceStandard: $2 input and $10 output per million tokens.
- A month
- $1,596a month
- 100 pull-request runs, 30 nightly runs and 20 deploys.
Then cap it. The runner counts judge calls and stops at the ceiling for its kind of run, and the check fails with a message that names the ceiling. A dataset that outgrows its budget shows up as a red check on the day it happens. The eval owner raises the ceiling with a reviewed commit.
Set timeout-minutes on the eval job as the backstop. GitHub’s workflow syntax cancels a job after 360 minutes by default. Set a limit that fits the eval, such as the 30 minutes in the example above. A concurrency group with cancel-in-progress lets a new push cancel the run still scoring the commit it replaced.
Wire the verdict into branch protection and deploys
Make eval-gate a required status check on main and set the rest of the rules around it. GitHub’s protected branch documentation and its deployment environment reference give the settings below, grouped by where they live. On a private repository, branch protection needs GitHub Pro, Team or Enterprise.
Workflow
- One job named eval-gate, unique across workflows
- GitHub warns that the same job name in two workflows makes required check results ambiguous and can block pull requests.
Branch protection on main
- Require status checks: eval-gate
- A required check passes when its status is successful, skipped or neutral. The gate job always runs and fails on a block.
- Expected source: GitHub Actions
- Anyone with write access can set a status. With the source selected, a status from any other person or integration does not allow the merge.
- A merge queue, or branches up to date before merging
- Either one re-checks the change against the latest main. Up to date means more runs, and a queue needs the merge_group trigger.
- Do not allow bypassing the above settings
- By default the rules do not apply to people with admin permissions. This setting applies them to admins too.
- Require review from Code Owners
- The owners of the evals folder approve every change to tolerances, the baseline, datasets, rubrics and the judge. Only approvals from people with write permission count.
Deploy workflow
- The deploy job needs the full eval
- Works on every plan. When a job fails, GitHub skips every job that needs it, so a block stops the deploy.
Environment: production, optional
- Required reviewers, with prevent self-review
- Up to six people or teams. One approval releases the deploy job, and the person who started the deploy cannot approve it.
- A custom deployment protection rule
- A GitHub App approves or rejects the deploy from the full eval run. GitHub lists these rules as public preview.
Decide who can override a blocked merge before the first block. With bypass turned off, an override is a commit to the evals folder, such as a new tolerance or an updated expected answer. List the eval owner and the product owner as code owners of that folder and require their review. Give the product owner write access, or that approval will not count. Every override then carries an owner’s approval, and the history records why.
Gate the deploy in its own workflow, which runs the full eval on each push to main.
on:
push:
branches: [main]
concurrency:
group: deploy-production
jobs:
full-eval:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v6
- run: ci/full-eval.sh evals/manifest.yaml
deploy:
needs: full-eval
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- run: ci/deploy.sh
The concurrency group runs one deploy workflow at a time, and a newer push replaces a run still waiting. Required reviewers and a custom rule add a person or an app on top of this gate.
The first month for a team of two to five
Illustrative plan: a team of two to five, the product owner included, adds evals to a product that calls a model. It describes no client engagement and no measured result.
- Week 1
Block on contracts and wire the gate.
Merge both workflows and make eval-gate a required check on main. Contract checks block from the first pull request, since they need no baseline. Quality metrics start in observe mode.
- Weeks 1 to 3
Build the baseline and the labels.
Nightly runs on main store each case’s output and score under one manifest. The domain expert labels 100 to 200 stored outputs, failing ones included.
- Week 4
Calibrate, pin and promote.
Score the labeled set with the judge and fix the rubric until both bars clear. Re-score ten stored nightly runs with the final rubric, and the eval owner pins them as the baseline. The product owner sets each tolerance, and each calibrated metric moves to compare.
One person can be the eval owner, a prompt author and the deployer, and the product owner can be the domain expert. Two pairs always need two people: the expert never labels outputs of a prompt they wrote, and a code owner who did not write an evals change approves it.
A team of two meets both rules when the second person is the product owner and the domain expert, with write access. If the team adds required reviewers with prevent self-review, that person also approves each deploy.
With these settings, a prompt change merges on evidence. Its contract checks passed, its quality scores held inside the tolerance the product owner set, and the run that showed it is pinned to a manifest. When a check fails, the run names the check, the manifest and the cases that moved.
ITECS builds and runs these gates as part of CI/CD pipelines for software with AI in it, one of its DevOps and platform engineering services.
