What are AI evals?
AI evals (short for evaluations) are automated tests for an AI system. Anthropic's engineering guide Demystifying evals for AI agents defines an eval as "a test for an AI system: give an AI an input, then apply grading logic to its output to measure success". You run the same set of tests after every change to the prompt, the model or the tools, and you compare the score with the previous run. You will also see two narrower terms: LLM evals for tests of a single model call, and AI agent evaluation for tests of a whole agent.
A simple eval sends one prompt and checks one answer. An agent eval is harder, because an agent works in many steps: it calls tools, changes files or records and reacts to what it finds. The guide splits an agent eval into a few parts:
- Task. One test with defined inputs and success criteria.
- Trial. One attempt at a task. Model outputs vary between runs, so each task gets several trials.
- Grader. The logic that scores part of the result. One task can have several graders.
- Transcript. The full record of a trial, including tool calls and intermediate results.
- Outcome. The final state of the environment after the trial.
- Evaluation suite. A set of tasks that measure one area of behaviour.
When you evaluate an agent, you test more than the model. In the guide's words, "we're evaluating the harness and the model working together". The same model can pass in one agent harness and fail in another.
In plain words
An eval is an exam for an AI system. You write questions where you know what a good answer looks like, and you grade the system every time you change something about it. Because an AI system can answer the same question differently each time, it sits each question more than once. The score then tells you whether your last change made it better, worse or left it where it was.
How an eval is graded
The grader is the core of every eval. The Anthropic guide names three kinds:
- Code-based graders compare strings, run unit tests or check the final state of a system. They are fast, cheap and reproducible. They are also brittle. A correct answer in an unexpected format fails.
- Model-based graders let a language model score the output against a rubric. This is LLM-as-a-judge. It handles open-ended answers, but the guide lists its weak points plainly. It is "non-deterministic", costs more than code and "requires calibration with human graders for accuracy".
- Human graders are experts who review the output. They are the most accurate and the slowest. Their main job in an automated suite is to calibrate the model graders.
Grade the outcome the agent leaves behind. Its last message can claim more than it did. The guide's example is a flight-booking agent that replies "Your flight has been booked". The outcome is "whether a reservation exists in the environment's SQL database".
Several trials per task raise a second question. What counts as a pass? The guide uses two metrics. pass@k is the chance that at least one of k trials succeeds. pass^k is the chance that all k trials succeed. They pull apart fast. With a 75% success rate per trial and three trials, the chance of passing all three is about 42%. For a tool where one good answer is enough, pass@k fits. For an agent that customers rely on every time, pass^k is the honest number.
Why evals matter
- You see regressions before users do. Without evals, the guide says, debugging is reactive: "wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed."
- Two kinds of suites answer two questions. Capability evals ask what the agent can do and start at a low pass rate. Regression evals ask whether it still does what it used to and "should have a nearly 100% pass rate."
- Evals speed up model upgrades. According to the guide, teams without evals "face weeks of testing" when a stronger model comes out, while teams with evals can "upgrade in days".
- You do not need hundreds of tasks to start. The guide's advice is that "20-50 simple tasks drawn from real failures is a great start."
How we use evals
For a large Czech industrial company we built an AI assistant that fills in technical parameters from customer inquiries. We measure it on an evaluation set of 50 real inquiries with 3,432 checked values. Two comparable runs three weeks apart showed parameter-level accuracy rising from 98.8% to 99.1%. The project also ships with 231 automated tests on real documents. They check all parameters of every item, so a value the model invents for a parameter nobody asked about counts as a failure automatically. The full story is in the case study.
We test our own headless agents, which run on a schedule with nobody watching them work, with eval suites too. Each case describes a situation an agent once got wrong or must never get wrong. One case puts an agent inside the hours when it must not send people non-urgent messages. It has just opened a pull request and is also due to send a weekly reminder. Three code-based graders read the decision file the agent writes. They check that the agent holds back both messages and leaves the reminder state untouched. Each case runs twice by default.
A gate script compares every run with a stored baseline. Among other rules, it fails when:
- any case tagged as security scores below 1.0,
- any case drops by more than 0.05 against its baseline,
- a case that has a baseline is missing from the run,
- the run is partial or a case ended with an error.
Only the agent's owner can move the baseline. A change that lowers a score cannot quietly become the new normal.
Agent evals are not cheap. Four of the cases shared by all our agents cost USD 6.64 at list prices for two runs each. The gate's default cost cap of USD 4 would cut that suite off before it finished, and the gate counts an incomplete run as a failure, so the shared suite runs with a higher cap. A person starts our evals before a release of the runtime our agents share, and whenever the owner asks.
What to watch out for
- A passing score can hide a broken grader. The guide is blunt: "You won't know if your graders are working well unless you read the transcripts and grades from many trials." Read failed transcripts. Some failures are the grader rejecting a valid answer.
- A strict grader can fail a better solution. Anthropic reports that Claude Opus 4.5 found a loophole in the policy of a flight-booking task on the τ2-bench benchmark. It "failed" the evaluation as written, but came up with a better solution for the user.
- A suite at 100% stops teaching you anything. "An eval at 100% tracks regressions but provides no signal for improvement," the guide says. Keep it as a regression suite and write harder capability tasks.
- Public benchmarks measure generic tasks. Your users care about your tasks, your data and your rules, so build your cases from your own failures.
- Evals cover only the scenarios you thought of. Real traffic brings the ones you did not, which is what LLM observability is for.
- Don't build a large suite too early. When you are still changing the agent every day, a few dozen cases from real failures give you most of the signal. Grow the suite as the failures you see get rarer and subtler.
Related articles:
- What is LLM-as-a-judge? – How a model-based grader works and where it goes wrong.
- What is an agent harness? – The part of the system an agent eval tests together with the model.
- What is LLM observability? – How you watch the cases your evals did not foresee in production.
- What is an AI hallucination? – The failure that negative tests in an eval are built to catch.
- What is prompt injection? – An attack that security-tagged eval cases check an agent resists.
- What are AI guardrails? – The runtime checks that complement what evals test before release.
Want to stay one step ahead?
Don't miss our best insights. No spam, just practical analyses, invitations to exclusive events, and podcast summaries delivered straight to your inbox.
