Your AI judge needs a performance review. Giving another model a clipboard does not make it management.
An LLM judge can help assess answers that are difficult to check with a simple rule. But its score is a measurement produced by another system, with its own failure modes. Before using that score to approve a release, test whether the judge recognises the mistakes that matter to your users.
The question is not whether a model can write a convincing explanation for a rating. It usually can. The question is whether that rating holds up when the answer is wrong, the wording is polished, or two reasonable people disagree.
Start with the decision behind the score
“Helpful” is a lovely adjective and a terrible test specification. Helpful to whom? A customer trying to recover an account needs something different from a reviewer checking whether the recovery process respected access boundaries.
Separate the dimensions. Factual correctness, useful next steps and respectful tone should not silently compensate for each other. A warm response that invents a refund policy should still fail the policy check. Customer service has enough problems without giving fiction a friendliness bonus.
Write the rubric around observable evidence. Specify what the judge receives, which source establishes the answer, and what counts as insufficient information. Where a condition is mechanically testable, check it directly. A judge should not guess whether a database record exists from an agent saying it created one.
Anthropic’s guide to agent evaluations distinguishes transcripts from actual outcomes and describes code-based, model-based and human graders. It also identifies human calibration as a requirement for model-based grading. That is a useful foundation, not a guarantee that any particular judge is ready for your workflow.
Let humans disagree before the model settles it
Build a calibration set containing clear successes, clear failures and genuinely awkward cases. Have reviewers label examples independently before seeing the judge’s verdict. Otherwise the model gets to chair the meeting before anyone has read the agenda.
Keep the original labels and reasons. If reviewers disagree, ask whether the evidence is missing, the rubric is vague or the product decision is genuinely contested. An adjudicated label is useful, but it should not erase the disagreement that produced it.
Consider an illustrative support answer that recommends escalation instead of an immediate fix. One reviewer values speed; another notices that the fix requires an unverified identity. The disagreement may expose a missing policy boundary. Changing the judge’s prompt will not resolve a decision the team has avoided making.
This is different from asking whether your suite covers enough behaviours, the concern in Evals are the product. Here, the test cases exist. We are checking whether the person, code or model marking them is measuring the intended thing.
Change the presentation, preserve the substance
The 2023 MT-Bench and Chatbot Arena study examined position, verbosity and self-enhancement biases in LLM judges. These are reasons to run local checks, not evidence that every current judge fails at the same rate.
For pairwise comparisons, swap the order of the same answers and check whether the preference survives. Test equivalent concise and verbose versions where appropriate, keeping the underlying facts intact. Record inconsistencies instead of choosing whichever run agrees with your favourite model.
A later systematic study of position bias, revised in November 2025, found variation across judges and tasks. One clean demonstration is therefore not enough to establish reliability across your product. Test the actual task types you intend to score.
Count the expensive mistakes separately
Report false passes and false failures against the reviewed reference labels. A false pass accepts a bad answer; a false failure rejects a good one. Those costs are rarely equal. Split the results by relevant task or risk category, and show the number of examples behind each result.
Keep a separate validation set out of rubric tuning. Repeatedly adjusting the judge against the same examples can produce a reassuring score that says little about unfamiliar cases. Small samples and unresolved labels should remain visible limitations.
My proposed workflow is to compare independent human labels with judge decisions, investigate disagreements, refine on the calibration set, then assess the frozen configuration on held-out examples. Treat that as a review method, not a universal certification threshold.
Version the judge model, prompt, rubric and dataset together. If any of them changes, rerun the relevant checks. As with AI change control, the evidence belongs to a configuration, not a reassuring name.
The judge can save people time. It should not save the team from deciding what good actually means.
