How to evaluate an AI agent before it touches production

Why 'it works in the demo' is not evaluation

The most common evaluation methodology we see in enterprises is informal: someone ran the system against a handful of examples, the outputs looked reasonable, and that was taken as sufficient evidence to move forward. This isn't evaluation — it's a spot check, and spot checks systematically miss the failure modes that matter most, because the person running them unconsciously tests the cases they expect to work. Before an AI agent is allowed to touch real customers, real employees, or real enterprise systems, it needs to be evaluated against a methodology defined in advance, covering the range of realistic inputs, and producing a repeatable score — not a gut feeling.

The five dimensions

A useful evaluation covers at least five dimensions, and treating any one of them as a proxy for the others is where most evaluation efforts go wrong: accuracy, reliability, safety, latency and cost. A system can score well on accuracy and still be unsafe to deploy. It can be fast and cheap and still be unreliable in ways that erode trust within a week. Each dimension has to be measured on its own terms.

Accuracy: defining 'correct' for this workflow

Accuracy sounds like the obvious first metric, but the harder question is what "correct" means for a specific workflow, and who gets to define it. A generic benchmark score for a foundation model tells you almost nothing about whether that model, embedded in your specific agent with your specific tools and your specific data, produces the right outcome for your specific process. Defining accuracy properly means sitting down with the people who currently do this work and building a rubric with them: what a fully correct output looks like, what a partially correct one looks like, and what's simply wrong. That rubric, not a leaderboard score, is what the agent should be measured against.

Reliability: behavior outside the happy path

Reliability is about what happens when the input isn't the clean, well-formed case the system was designed around. Real users misspell things, ask incomplete questions, provide contradictory information, or ask for something the system was never built to do. A reliable agent needs to behave predictably in all of those cases — which, most of the time, means recognizing when it's uncertain and saying so, rather than producing a confident, plausible-sounding wrong answer. Testing for reliability means deliberately constructing an adversarial and edge-case test set, not just a set of examples the system is expected to handle well.

Safety: worst case and blast radius

Safety evaluation starts from a different question than accuracy: not "how often is this right," but "what's the worst thing that happens when it's wrong, and how far does that damage spread." An agent that drafts an internal report and gets a fact wrong has a very different blast radius than one that can issue a payment or change a customer's account permissions. The higher the blast radius of an action, the more evaluation and human oversight it needs before the agent is allowed to take it autonomously — this is also where prompt injection and tool-misuse testing belong, treating the agent's inputs as adversarial rather than as ordinary user requests.

Latency and cost at real volume

A system that performs well against a hundred manually run test cases can behave very differently at the volume and concurrency of actual production traffic. Latency compounds across multi-step agent workflows in ways that aren't obvious from a single test run, and cost scales with usage in ways that can quietly turn a viable pilot into an unaffordable production system. Evaluating latency and cost means load-testing against realistic volume assumptions before committing to an architecture, not discovering the numbers after launch.

Building an evaluation harness before deployment

All of this needs to be automatable, because it needs to be re-run every time the model, the prompt, the tools, or the surrounding logic changes — which, in an actively developed agent, is often. That means investing in an evaluation harness: a versioned set of test cases mapped to the rubric described above, a way to run the agent against that set automatically, and a way to compare results across versions so a change can be shown to be an improvement rather than assumed to be one. Teams that skip this step end up making every change on faith, which is precisely the situation evaluation is meant to prevent.

Evaluation doesn't stop at launch

The test set that was representative at launch stops being representative as the business changes, as usage patterns shift, and as edge cases the initial evaluation didn't anticipate start showing up in real traffic. Production agents need continuous evaluation: sampling real interactions, routing a subset for human review, and feeding confirmed errors back into the test set so the evaluation harness gets stronger over time instead of going stale. Without this loop, the accuracy of a production agent tends to drift downward exactly when nobody is watching closely, because it already passed its one-time evaluation months earlier.

Who owns evaluation

Evaluation needs an owner, and it usually shouldn't be the same team that built the agent, for the same reason engineering and QA are usually separate functions: the incentive to ship works against the incentive to find problems. In practice this can be a shared responsibility between the engineering team and whoever owns the business process the agent supports — but it needs to be someone's explicit job, with the authority to block a release if the evaluation results don't clear the bar.

Evaluating against a moving target

Foundation models are updated by their providers on a schedule enterprises don't control, and prompts, tool definitions and retrieval sources inside the agent itself change as the team keeps improving it. Every one of those changes is a potential regression, even when it's intended as an improvement — a prompt rewrite that fixes one failure mode can easily introduce another, and a model upgrade that improves general reasoning can shift behavior on the specific edge cases your rubric was tuned against. Treating the evaluation harness as versioned alongside the agent itself, and re-running the full suite on every change rather than just the change that prompted the update, is what catches these regressions before they reach production rather than after a customer or employee does.

The role of human review sampling

No automated rubric captures everything a human reviewer would catch, particularly for judgment-heavy outputs where "correct" has some genuine ambiguity. A practical middle ground is structured sampling: routing a defined percentage of production interactions — weighted toward the cases the agent flagged as uncertain, plus a random baseline sample — to a human reviewer on a regular cadence, with a simple structured way to record disagreements with the agent's decision. This does two things at once: it catches drift and edge cases the automated harness doesn't cover, and it generates exactly the labeled examples needed to expand the test set over time. Skipping this step tends to produce evaluation suites that stay static while the real traffic distribution moves further away from them every month.

What 'good enough' looks like at different stages

The evaluation bar should not be identical for an internal tool used by a handful of trained employees and a customer-facing agent handling financial transactions — but the difference should be a deliberate calibration, not an excuse to skip evaluation for the lower-stakes case. For a low-risk internal tool, a smaller test set, a lighter sampling cadence, and a shorter list of guardrails may genuinely be sufficient, provided that bar was set consciously rather than by default. For anything customer-facing, financially consequential, or operating with broad system permissions, the bar needs to be materially higher on every one of the five dimensions before the agent is allowed to run unsupervised. Enterprises get into trouble not when they calibrate the bar to the risk, but when they never calibrate it at all and let every project inherit whatever evaluation happened to get done — or skipped — on the last one.

A practical checklist

Before an agent goes into production, we expect clear answers to: What's the rubric for correct, and who defined it? What does the adversarial and edge-case test set cover? What's the worst-case action this agent can take, and what happens if it takes it wrongly? What's the latency and cost at expected production volume, not test volume? Is there an automated harness that reruns on every change? And is there a continuous evaluation loop feeding real production data back into that harness? If any answer is "we haven't done that yet," the agent isn't ready, no matter how good its outputs look today.

Closing

Evaluation is not a gate you pass once before launch — it's an ongoing discipline that has to be built into the system from the start and maintained for as long as the agent is in production. The organizations that treat it this way are the ones whose AI systems are still trusted, and still working, a year after launch. The ones that treat evaluation as a formality tend to find out the hard way what it was for.