Evaluation in the AI SDLC: Your AI Passes the Tests, But Is It Actually Working?

There is a kind of failure that never announces itself. The tests pass, the schema validates, nothing throws an exception, but the feature is wrong anyway.

That happens because of what "done" has always quietly meant, which has now changed.

What done used to mean

Ask a developer when a piece of work is finished and you get a version of the same list: the code runs, the tests are green, review is approved, it survived staging. Underneath all of that sits one check doing most of the real work. Did the thing return the shape it was asked for?

A string came back. The object parsed. The value was one of the three options declared. For a function that returns the same answer to the same question every time, that check sits so close to correctness that most of us stopped noticing the gap. The shape stood in for the substance, and it stood in well.

None of that was sloppy, it was true for a long time, and the tooling built on top of it is genuinely good.

The contract still passes

Now put a language model in the middle of that pipeline.

The output still matches the schema. A classifier constrained to three categories still returns one of the three, because you constrained it to. An agent still emits a well-formed tool call. Every test in the suite goes green and the deploy goes out.

The contract is met. But what the contract was expected to be standing in for is not.

Two places the decision goes wrong

Take zero-shot classification, for instance, a popular method used in many AI features shipped in the last two years. A user types something, a model reads it, and the model analyses and returns a, b or c.

You can see it working. The outputs are always in your option set, never null, never a paragraph of prose where an enum belonged. So the part you can see is fine, and two things you cannot see are not:

• Whether it returned a when the right answer was b.
• Whether your instructions and the context you fed it were even sufficient for a correct answer to be reachable.

That second one is the sharper of the two and it gets asked least. A model can be given a task it cannot do from the material provided, and it will still answer, confidently, in the right format.

The same scenario appears with tools. Give an agent four tools and it will produce well-formed calls to them. Well-formed is not the question. The questions are whether it should have reached for a tool at all, and whether it will reach for it again next time on the same kind of input. Neither is expressible as a predicate over a single run.

Nothing catches it, and that is the expensive part

Here is what the absence looks like when you measure it.

I have seen a model sit at roughly 80% accuracy in experiments and drop to around 65% in production. A standing gap of 15%, with no incident, no alert and no red build. It was simply the cost of the two environments not being the same, and nobody had a number that made it visible.

When that model did degrade, the first places anyone looked were all inside the model. It turns out that the actual cause was upstream: a large proportion of the references used for matching carried invalid identifiers and were quietly poisoning the results. The fix was one filter check and it re-established the performance. Without evaluation built into the experiments, there was no way to even know the production performance was supposed to be higher in the first place.

I am not offering that as proof, but it should be persuasive enough for building the measurement before the urgency kicks in. The contrast is apparent, evaluation saves you the cost needed to fix hidden technical debt that surfaces in production, although it leads to a longer time-to-market.

Another project had a subtler failure: payloads triggered by hand, and responses extracted manually into a shared document for human review. Replacing it with a harness that combines human expert review and LLM judgements with evaluation metrics removes the need for fussy manual work.

The gate that replaces it

When people say "evaluation" they might mean an offline scoring run over a fixed defined dataset, an online guardrail checking outputs in production, the general activity of knowing whether the system works, or a combination of any of these. They are different things with different costs and complexities. The first one is most common: a repeatable scoring run over a fixed set of cases, producing a number you can compare across changes. The online guardrail is a separate and usually later problem.

So the new gate has to ask something the old one never had to: Did it make the right decision most of the time, across various cases and scenarios?

In practice that means four things:

• A scenario set, not a single call. One request proves nothing about a non-deterministic LLM call component. You need a collection of cases you can rerun.
• An LLM-as-a-judge, plus a human expert where judgement is genuinely required. A model can score most outputs against criteria. A subject matter expert is still the source of what "good" means.
• A binary verdict with a reason attached to every score. Pass or fail, with the reason recorded, so a failure is diagnosable rather than just countable. Critical failures should drive the overall verdict rather than being averaged away.
• The expert's judgement distilled into reusable criteria. This is the part that compounds. Every time you convert "I know this one is bad because..." into a written criterion, the next run needs less of the expert's time.

That last point answers the objection that decision makers might have, which is that evaluation is expensive ceremony that slows delivery. Yes, the setup does cost time, but the more of the expert's decision procedure you capture as criteria, the cheaper every subsequent evaluation gets, and you can be confident in the performance.

How much evaluation is enough

The uncomfortable answer to "when do we need evaluation" is that I cannot think of a live AI feature that needs none. The question is not whether we need one. Instead, it is how much effort we should put into evaluation, and that decision comes down to two questions.

How precise does the answer have to be? If there is flexibility in wording, length or form, and the feature is simple, then create a golden dataset of 10 scenarios and review the responses yourself. That is also a form of simple evaluation, as long as the scenarios are properly chosen. If it is an AI system that leaves no room for misjudgement, it needs a proper evaluation pipeline.

How wild can the input get? A narrow, well-behaved input space gives you few cases to check. A feature taking open user input (say, a chatbot) does not let you assume its users are sensible. Some will enter nonsense for entertainment. Some will be hostile and probing for prompt injection. Each of those is an edge case your system would meet in production, and therefore needs to be properly evaluated.

As an example, for an extraction service that assembles structured information from user inputs arriving over time, faithfulness to the source is compulsory. There is little to no room for misjudgement, and it got a proper evaluation harness. Production traces sampled into development and test sets with balanced coverage and no overlap between them, questions generated from the material, and a declarative harness scoring the answers.

Where it sits in the cycle

Here is what it takes for evaluation to sit inside the development cycle:

• Evaluate on a curated golden dataset. If the system is live, curate the set from anonymised or processed production data. If the system is not live yet, or there is no usage worth pulling from, mock up the scenarios you expect users to put through it.
• Run the same eval pipeline on the production branch before release. Experiment-production parity is what closes the performance gap. A harness that only runs on an experimenter's laptop measures a system nobody is using.
• Ship small spikes with evaluation attached. The evaluation is part of the change, not a follow-up ticket that gets deprioritised.
• Structure production code so experiments can reuse rather than replicate. Otherwise the experiment code reimplements what production already does, and the two drift. Worst case, the drift is invisible until production performance drops.

One caveat, since I have been arguing for rigour. I once made the mistake of scoring a multi-step pipeline as a single unit, so a problem in any one step disappeared into the overall score. Mechanically complete and analytically useless. Captured and scored well are not the same thing, and a harness with an unexamined blind spot can be more dangerous than no harness at all, because it produces a number people believe.

The definition of done did not need replacing. It needed one more line, and the line is a measurement.