How to evaluate an AI automation before you trust it

You've built an AI step into a workflow. Maybe it reads supplier emails and pulls out invoice numbers, or sorts support tickets into queues. You tried it on a handful of examples and it looked right every time. Now someone asks the obvious question: can we switch it on and stop checking?

A few good-looking runs can't answer that. Language models don't behave like ordinary code. As OpenAI's evaluation guide puts it, "Models sometimes produce different output from the same input", and evaluations are how you test a system despite that variability. The same guide lists "vibe-based evals" (treating "it seems like it's working" as the strategy) as an anti-pattern. Here's a practical way to do better without a research team.

Decide what "working" means before you test#

Most automations fail their first evaluation for a boring reason: nobody wrote down what a correct result looks like. Anthropic's guide to defining success criteria asks for criteria that are specific, measurable, achievable and relevant. Its example of specific is swapping "good performance" for "accurate sentiment classification".

For a business workflow, translate that into the decisions the automation makes. For the invoice example, it might be:

• The invoice number, supplier name, due date and total must match the document exactly.
• If any of those fields can't be found, the output must say so rather than guess.
• Emails that aren't invoices must be routed to a person, not processed.

Then decide what each kind of mistake costs. A wrong queue for a support ticket wastes a few minutes. A wrong bank detail on a payment is a different category of problem. OpenAI's guide recommends a pass/fail threshold alongside any numerical score, and that's the right instinct. Agree the bar with whoever owns the process before you see any results, so the numbers can't talk you into lowering it.

Build the test set from real work#

Your test cases should look like the work the automation will actually see. Anthropic's guide says to "mirror your real-world task distribution" and not to forget edge cases. OpenAI's goes further and asks for typical, edge and adversarial cases, and suggests you log as you develop so you can mine those logs for new test cases. It also names datasets that don't reproduce real production traffic as a design mistake.

In practice, pull a few months of real inputs: the emails, forms or documents the process already handles. Have the person who does the job today record the correct answer for each one. That labelled set is your ground truth, and it's worth more than any prompt tweak.

Deliberately include the awkward cases you already know about. Scanned PDFs. Invoices with two totals (before and after tax). A reply-all thread where the invoice is three messages down. An email in a language your team doesn't usually see. If you're wiring the model up to email or the web, add a few inputs that try to give it instructions, since that's how prompt injection reaches automations.

How many cases? Anthropic's guide favours volume: "More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." Ten examples tells you almost nothing. A few hundred real ones starts to tell you which mistakes happen often and which are rare.

Grade with code wherever you can#

If you have to read every output by hand, you'll run the evaluation once and never again. Anthropic recommends structuring tests so grading can be automated, using things like multiple choice, string matching, code-based checks or a model as the grader.

A lot of automation output is easy to check with plain code. Did the extracted invoice number match the label exactly? Is the total the same number once you strip currency symbols? Is the output valid JSON with every required field? Did the ticket land in the right queue? These checks are fast, free and don't drift.

Keep a score per field and per case type, not a single overall figure. "Correct on clean PDFs, weak on scanned ones" is something you can act on. One average hides it.

Using a model as the judge, carefully#

Some outputs can't be checked with string matching: a drafted reply to a customer, a summary of a contract clause. Here people often use a second model to grade the first. That can work, but the judge has its own failure modes.

A 2023 study, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng and colleagues, found that strong model judges reached over 80% agreement with human preferences, about the same level of agreement humans had with each other. The same paper documents position bias (favouring whichever answer comes first), verbosity bias (favouring longer answers) and self-enhancement bias. One fix the authors use for position bias is simple: call the judge twice with the answers swapped, and only count a win if the same answer is preferred both times.

OpenAI's guide gives similar advice: prefer pairwise comparison or pass/fail over fine-grained scores, control for response length, and have the judge reason before it scores. Anthropic's example grading code notes that it's generally best practice to use a different model for grading than the one that produced the output.

Most importantly, check the judge against people. Have a human grade a sample, compare, and only rely on the model judge once it agrees with your reviewers. OpenAI lists "not calibrating your automated metrics against human evals" as another anti-pattern. A judge you haven't checked is just a second unverified model.

Read the failures, not just the score#

A score of 94 out of 100 tells you how often it fails. It doesn't tell you whether you can live with the six failures. Open every failed case and sort them into groups: missed field, wrong field, made-up value, wrong routing, malformed output.

Made-up values deserve special attention. NIST's Generative AI Profile calls this confabulation, where systems "generate and confidently present erroneous or false content". For an extraction task, an empty field that gets flagged is an inconvenience. A plausible invented due date that flows into your accounts system is the failure that hurts, because nobody notices it.

The same NIST document warns against extrapolating performance from "narrow, nonsystematic, and anecdotal assessments", and says to document the limits of any benchmark relative to the environment you'll deploy in. So be honest in your write-up about what the test set doesn't cover. If it holds no handwritten forms, you haven't tested handwritten forms.

Decide where people stay in the loop#

Evaluation results should shape the workflow, not just produce a yes or no. Usually the answer is a mix. Let the automation handle the case types where it's reliable, and send the rest to a person: low-confidence outputs, missing fields, amounts above a set limit, anything that doesn't look like the inputs you tested.

A sensible way to go live is to run the automation in parallel with the existing process for a few weeks. The person still does the work, the automation's answers are logged, and you compare the two. Disagreements become new test cases. Once the comparison matches what your evaluation predicted, start removing the human step one case type at a time.

Keep testing after launch#

An evaluation isn't a one-off sign-off. OpenAI's guide recommends continuous evaluation: run your evals on every change and grow the eval set over time. "Every change" includes changes you didn't make. A provider updating or retiring a model, a supplier redesigning its invoice template, or someone tweaking the prompt can all shift results.

Keep the test set and grading scripts in your own repository as plain files rather than locked inside one vendor's dashboard. Tools come and go. The same OpenAI page notes, as of October 2026, that OpenAI is deprecating its hosted Evals platform, which becomes read-only on 31 October 2026 and is scheduled to shut down on 30 November 2026. A folder of labelled real examples and a script that scores them will outlive any of that.

If you do one thing this week, collect 50 real inputs, have the person who does the job label the right answers, and run your automation against them. You'll learn more from reading those results than from any number of demos.

Sources

1. Evaluation best practices — OpenAI API documentation (checked 2026-10-11)
2. Define success criteria and build evaluations — Anthropic (Claude Developer Platform documentation) (checked 2026-10-11)
3. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, Zhang, Gonzalez and Stoica (arXiv, 2023) (checked 2026-10-11)
4. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) — National Institute of Standards and Technology (July 2024) (checked 2026-10-11)