Most AI pilots are judged on impressions. A team runs a few dozen examples, the outputs look reasonable, a demo goes well, and the project moves forward. With agents, that approach breaks down quickly. An agent takes many steps, calls tools, and changes things. A fluent final message can hide a wrong database update, a skipped check or a task that was never finished.
If you are going to trust an agent with real work, you need evaluation that checks outcomes. This post describes how we approach it.
Evaluate the outcome, not the transcript
The most important shift is from judging what an agent said to judging what it did. For each task, define the end state that counts as success, and check that state directly.
- A coding agent succeeds when the change compiles, the relevant tests pass and no unrelated tests break.
- A data agent succeeds when the query returns the correct rows, checked against a known answer.
- A document agent succeeds when each extracted field matches the source, and each claim cites the passage it came from.
- An operations agent succeeds when the incident is resolved and the metrics that triggered it return to normal.
These checks are ground-truth verifiers: deterministic programs that inspect the result and return pass or fail, with a reason. They are not perfect, but they are repeatable, cheap to run and hard to talk your way past.
When ground truth is hard to get
Some outcomes cannot be checked by a program. Is this summary faithful? Is this customer reply appropriate in tone? For these, teams often use another model as a judge. That is reasonable, with three conditions:
- Calibrate the judge against people. Have domain experts grade a sample, and measure how often the model judge agrees. If agreement is low, the judge is not ready.
- Use narrow rubrics. "Rate quality from one to ten" invites noise. "Does the reply contain any commitment not present in the policy document?" is far more reliable.
- Keep people in the loop for drift. Re-check a sample of judge decisions on a schedule, because both the judged model and the judge can change.
Where you can, break a fuzzy outcome into checkable parts. Faithfulness, for example, can be partly verified by confirming that every cited passage exists and supports the claim attached to it.
Build an evaluation harness
An evaluation harness is the system that runs your agent against a test set and records the results. A useful one has these properties:
- Realistic tasks. Drawn from real work, with sensitive details removed, and covering common cases, edge cases and known failure modes.
- Reproducible environments. Each run starts from the same state: the same files, records and tool responses. Sandboxed copies of real systems work well.
- Full traces. Every step, tool call, input and output is recorded, so failures can be diagnosed rather than guessed at.
- Multiple runs. Agents are not fully deterministic. Run each task several times and report the pass rate, not a single result.
- Cost and time. Record tokens, tool calls and elapsed time per task. An agent that succeeds slowly and expensively may not be worth deploying.
- Safety cases. Include tasks with injected instructions, missing permissions and ambiguous requests, and check that the agent refuses, escalates or asks rather than guessing.
Gate releases on results
Evaluation only protects you if it is wired into how changes ship. Every change to the model, the prompts, the tools or the retrieval data can change agent behavior. Treat each one like a code change:
- Run the evaluation suite automatically before promotion.
- Set thresholds per task category, and block promotion when a category drops below its threshold.
- Require a named person to approve any exception, with the reason recorded.
- Keep the history, so you can see whether quality is trending up or down over time.
This is a regression gate. It turns evaluation from a report people may read into a control that acts on its own.
Close the loop from production
No test set anticipates everything. Production will surface failures you did not imagine. The discipline that matters most is simple: every confirmed production failure becomes a new test case. Over time, your evaluation suite becomes a record of everything that has gone wrong, and a guarantee that it will not go wrong the same way twice without someone noticing.
In production, the same verifiers can run continuously on a sample of live work. When a verifier fails, the system can hold the result for review, route it to a person, or roll back to a previous configuration, depending on the stakes.
Report what matters to the business
Technical pass rates are necessary, but they are not what leaders need. Translate results into terms tied to the workload: tickets resolved without rework, documents processed without a correction, incidents closed within target time, cost per completed task. These are the measures that justify widening an agent's scope, or narrowing it.
A starting checklist
- Write down the success state for each task the agent performs.
- Build a verifier for every success state that can be checked by a program.
- Assemble a test set of at least a few dozen realistic tasks, and grow it.
- Calibrate any model-based judge against expert grading.
- Wire the suite into your release pipeline as a blocking gate.
- Route production failures back into the test set.
Verifiable evaluation is how we build our own products, including TuQuant Medha for the software lifecycle, and how we build agents for clients through our agentic AI development practice. If you want a second opinion on how your agents are measured, contact us.