How to test an AI system before production: evals for business teams

Most AI projects I am asked to review were approved after a demo. Someone typed ten questions, the answers looked good, and the project got a budget. Nobody could say how often the system was right on the cases that actually arrive on a Monday morning. An eval answers that question with a number before launch. Below is the process I use, with figures from systems I built that are running in production.

What is an AI eval, in plain terms?

An eval is a fixed set of real cases, each with the answer you expect, plus a rule that says when an output counts as correct. You run the AI system on every case, count the passes and compare the score with a threshold agreed in advance. Run it again after every change, and the same set tells you whether the change helped or hurt.

A demo shows that a system can produce a good answer. An eval measures how often it does, on your data, under your definition of correct. For a manager the eval score is the number that goes into the go or no-go decision, the same way a defect rate goes into a decision about a production line.

How do you test an AI system step by step?

Where should the test cases for an AI eval come from?

Take them from your own history. Cases written for the test come out clean and short, while real ones contain typos, three requests in one email, a scan photographed at an angle. If the system will read help desk tickets, take past tickets. If it will read documents, take documents that already went through the manual process, so the correct values are known.

Often the labels already exist. At a major Italian bank, the help desk had a history of tickets with the category a person had assigned. That history became the reference for an LLM agent that now routes more than 1 million tickets a year across 400+ categories. A labelled history like that is one of the prerequisites I check before agreeing to start a routing project.

Pick cases by sampling, then add the hard ones on purpose: rare categories, long documents, the customer who writes in dialect. A test set made only of easy cases produces a high score that will not survive the first week in production.

How strict should the scoring be?

As strict as the person on the receiving end. For Estyl, a conversational fashion assistant searching a catalogue of more than 2 million products, a response counted as correct only if every item in it matched the request: size, colour, price, stock. One wrong item made the whole answer wrong, because that is how the shopper sees it. Under that rule over 90% of responses were fully correct, and over 95% had at most one or two wrong outfits. The Estyl case study describes the retrieval work that moved that number.

A lenient rule, such as counting the share of correct items across all answers, would have produced a nicer number and hidden the answers that disappoint users. I report both, and decide on the strict one.

What should an AI system's score be compared against?

Against the people or process doing the job today, measured on the same cases with the same rule. On the bank help desk, the LLM agent reached 81% routing accuracy against a human baseline of 75% on the same task, and that comparison made the case for going live. After launch, the metric the bank already tracked moved too: average resolution time fell from 5-6 days to 2-3, because fewer tickets bounced between the wrong teams. The ticket routing case study has the details.

If nobody has measured the current process, that is the first job. Ask two or three experienced people to label the same sample independently and see how often they agree. A target of 95% makes no sense on a task where your own experts agree 80% of the time.

What score is good enough to go live?

It depends on what a wrong output costs and on what happens to the cases the system is unsure about. A system that sends uncertain cases to a person can go live at a lower automatic accuracy than one that acts alone, because the errors it would make are caught first.

Validation checks make that routing concrete. In a pipeline that extracts data from more than 2,500 financial documents a day, each document goes through about 30 checks, such as totals that must reconcile. Only documents that pass every check are cleared without a person; the rest go to an analyst with the failing check named. Manual work fell by 84%. The go-live question there was how many documents cleared automatically with no error slipping through.

How do you stop a change from breaking what already worked?

Rerun the full eval after every change: a new prompt, a different model, a new retrieval setting, an updated category list. LLM systems break without raising an error. A prompt tweak that fixes one complaint can break twenty cases nobody was looking at, and the only way to see it is to score the whole set again and compare.

Hamel Husain describes the cheapest layer of this as unit tests: fast, code-level assertions that run on every change, with human review of outputs on top (Hamel Husain, March 2024). In practice I keep the eval in the same repository as the code and treat a drop in score like a failing test: the change does not ship until the drop is explained.

What should you monitor after an AI system goes live?

Production cases drift away from the test set. New products, a new form, a policy change, and the cases look different from the ones you tested. Three signals cover most of it. The first is how often validation checks fail or uncertain cases go to people; a rising rate means inputs changed. The second is corrections: every time a person fixes an output, that is a labelled case. The third is a small random sample of live outputs reviewed every week or month by someone from the business.

Feed the failures back into the eval set. After a few months the test set reflects production better than the one you launched with, and every later change is measured against it.

Which mistakes make an AI eval worthless?

The first is testing on invented or cleaned-up examples, which produces a score that production will not match. The second is choosing the scoring rule after seeing the results. The third is reporting a single average that hides the failures on rare but expensive cases; break the score down by category or document type. The fourth is skipping the human baseline, so nobody knows whether 85% is good. The fifth is running the eval once before launch and never again, which means the first regression is found by a customer. The last is leaving the eval to the developers alone. The people who do the work today know what a correct answer looks like, and the business owner should sign off on the threshold.

For how I design agents with evaluation and guardrails built in from day one, see my AI agent development service.