A driving test conducted entirely in an empty car park might produce excellent pass rates. I would still like to know whether anyone tried a roundabout.
AI agent evaluation has the same problem. A green score tells you what happened inside the test you designed. It cannot quietly inherit all the questions you forgot to ask. Before celebrating the percentage, I want to see the coverage: which behaviours, under which conditions, with what evidence?
My starting point is simple: keep passed, failed and not tested visibly separate. That third state deserves a seat in the release meeting. Preferably before the slide with the confetti.
A successful task can leave two questions unanswered
Consider an illustrative booking assistant. It creates the requested reservation, and the evaluation checks the database. Good. That establishes an outcome. It does not, by itself, establish that the assistant respected the user’s permitted scope, or that it handles an interrupted booking safely.
Those require different observations. For authority, inspect the attempted actions and the enforced access decisions. For recovery, introduce an interruption and inspect the resulting state, including whether the user receives an honest explanation. A final answer saying “all done” is a claim to verify.
Anthropic’s guide to agent evaluations distinguishes tasks, repeated trials, graders and outcomes. It also explains why different graders can examine different aspects of a run. That distinction matters: a correct-looking transcript and a correct environment state are not interchangeable evidence.
Choosing what to test is a product decision. That is why evals are part of the product itself. Coverage makes that decision inspectable. It exposes what the reassuring headline leaves out.
Build a coverage map before a bigger dashboard
I would start with one important workflow and three questions: did it achieve the intended result, stay within authority, and recover appropriately when something went wrong? Then choose conditions that could change the answer. Normal operation, revoked access and a tool timeout are useful starting points for our booking example.
This is a proposed working method, not a universal benchmark. Each meaningful cell needs a test case, an expected behaviour, evidence to inspect and an owner. Some combinations may be irrelevant; label those with a reason. “Not applicable” should not become a polite hiding place for “we haven’t built that test”.
A coverage map is useful precisely because it resists the urge to average everything. A product might complete ordinary bookings reliably while its access-revocation behaviour remains untested. Combining those into one reassuring number hides a decision the team still has to make.
Nor does every failure mean the agent is broken. A test environment can fail, or a grader can reject an acceptable result. Keep an invalid or inconclusive run distinct from an agent failure, retain the trace, and investigate before changing the release decision.
One successful run is a rather small sample
Coverage and repeatability answer different questions. You can cover many situations once and still miss inconsistent behaviour. You can repeat one easy situation hundreds of times and still learn nothing about revoked access.
The original τ-bench research evaluated tool-using agents against policies and end-state goals, and introduced pass^k to examine consistent success across repeated trials. The useful lesson here is the measurement question, rather than its historical model rankings: how often does the workflow keep succeeding when you repeat it?
ReliabilityBench, a research preprint, separately examines repeated execution, equivalent prompt variations and controlled tool failures. That offers another useful lens for designing stress cases. It is not a certificate that your deployment is ready.
Make the release claim smaller and more useful
Instead of “the agent passed evaluation”, write down the supported claim: the workflow, configuration, tested conditions, number of trials and observed results. Name the gaps beside it. Keep the model, prompt, tools and test environment versions with the evidence so a later change does not silently inherit an old pass.
For high-impact actions, an untested authority or recovery behaviour may justify holding back that capability. For a read-only prototype, a restricted pilot may be reasonable. These are risk decisions for the responsible people, not something a green badge can make on their behalf.
This also changes the product manager’s job around AI agents. Ask which real user problem the tests represent, and what happens to that user when the workflow cannot finish. Otherwise we are optimising the exam paper while the customer is still waiting for a booking.
At the next review, put the untested cases on the first slide. Then decide which one matters enough to test next.
