Agentic Systems: The Eval Harness You Already Have

This is Part 3 of Agentic Systems: The Decision Layer, the first series in a pillar on building agentic systems.

• Part 1: Orchestrators and Specialists: how many agents, what each owns, and the contract between them.
• Part 2: Autonomy You Have to Earn: routing a decision to a human or to execution, and why that gate has to be earned per decision.
• Part 3: The Eval Harness You Already Have: scoring specialists against outcomes, and feeding that score back into the system.

The first two posts in this series both ended with a dependency on something not yet built. Part 1 said an orchestrator should weigh specialists by their calibrated accuracy. Part 2 said a decision type earns autonomy by demonstrated reliability over a window. Both assume something is measuring reliability honestly and continuously. That something is the subject of this post, and the good news is that most systems are already emitting the raw material for it.

Table of Contents

• Tests Check Code, Not Judgement
• Three Parts, One of Which Is Usually Missing
• Calibration Is Not Accuracy, and It Matters More
• Bucketing: One Number Is Still Too Coarse
• Offline Evals and Online Calibration Are Different Tools
• The Outcome Lag Problem
• Correlated Error and Gaming
• The Honest Bill
• When NOT to Build One
• Conclusion

Tests Check Code, Not Judgement

A unit test asserts that given this input, the function returns that output. It is a statement about determinism, and it is exactly the right tool for the code around an agent: the parsing, the routing, the schema validation, the retry logic.

It is the wrong tool for the judgement inside one. The severity specialist from Part 1 is not wrong in the way a function is wrong. Its code executes correctly, its output validates against the schema, and it returns a well-formed severity assessment with a confidence attached. The assessment is simply not very good, for this class of alert, this month. No assertion catches that, because there is no fixed correct answer to assert against, and the thing that changed is not in the code at all.

This is the gap an eval harness fills, and the reason it is not optional infrastructure for a system that routes on reliability. What it actually is, stripped of vendor framing: a mechanism that compares what the system predicted against what actually happened, aggregates that comparison, and feeds the result back into the system's own behaviour. No LLM judge is required. No platform is required. It requires an outcome you can eventually observe and the discipline to write down a prediction before you know whether it was right.

Three Parts, One of Which Is Usually Missing

The minimum viable version has three components, and teams reliably build the first two and stop.

Record the prediction. At decision time, persist what each specialist said, with its confidence and enough input context to make sense of it later. Most systems already do this, because it is also what debugging requires.

Record the outcome. When reality resolves, persist what actually happened, joined to the prediction. Many systems do this too, in a metrics pipeline or a data warehouse, because it is also what reporting requires.

Feed it back. Compute reliability from the joined record and let it change how the system behaves - as a weight in the orchestrator, as an input to the promotion and demotion rules from Part 2. This is the step that is usually missing, and its absence is the difference between a dashboard and a harness.

The distinction matters more than it sounds. A calibration figure that a human reads in a weekly review is a report. It informs someone. A calibration figure wired into the orchestrator's weighting changes the next decision automatically, which means the system responds to its own drift without waiting for anyone to notice. The wire from the score back into the decision is the harness. Everything upstream of it is just data collection.

The trading system from earlier in this series closes that loop concretely: every signal, decision, trade, and outcome is persisted, a calibration job aggregates those outcomes by researcher and by confidence bucket into accuracy rates, and the synthesizer uses those rates as weight multipliers when combining signals. A researcher's influence on tomorrow's decision is a direct function of its own resolved track record rather than how confident it sounds today.

Calibration Is Not Accuracy, and It Matters More

Here is a distinction that changes how you build the aggregation step, and it is routinely collapsed.

Accuracy asks how often a specialist is right. Calibration asks whether its confidence means anything. A specialist that says "70% confident" and is right about 70% of the time on those calls is well calibrated, even though it is only right 70% of the time. A specialist that says "95% confident" and is right 70% of the time is more accurate than its calibration suggests on some calls and dangerously wrong on others - and for an orchestrator, that is the worse failure.

The reason is what the orchestrator does with the number. It is not choosing one specialist's answer; it is weighing several against each other. To do that it needs the confidences to be comparable and meaningful - a 0.9 from the severity specialist has to mean roughly the same thing as a 0.9 from the ownership specialist, or the weighted combination is arithmetic performed on incommensurable units. A confidently wrong specialist does more damage in a panel than an accurately uncertain one, because the aggregation trusts the stated confidence.

This also explains why the structured scored contract from Part 1 insists on a confidence field rather than just a verdict. Without it there is nothing to calibrate, only accuracy to count - and accuracy alone tells the orchestrator how often to believe a specialist on average, not how much to believe this particular claim.

Bucketing: One Number Is Still Too Coarse

Part 1 argued that a single agent's aggregate accuracy hides its per-competency distribution. Exactly the same failure recurs one level down, inside a single specialist, and the fix is the same.

A severity specialist at 88% overall may be at 96% on alert types it has seen thousands of times and 54% on services deployed in the last month. Pooled, those become one number that describes neither case and misleads in both directions: it understates trust where trust is warranted and overstates it exactly where the system is weakest.

So reliability is computed per bucket, not per specialist, where a bucket is whatever dimension the specialist's performance actually varies along. Confidence level is almost always one - a specialist's high-confidence calls and low-confidence calls deserve separate scores, which is precisely what calibration requires. Beyond that the buckets are domain-specific: alert class, service maturity, time of day, market regime.

The practical constraint is sample size. Every additional bucketing dimension divides the data, and buckets that are too fine contain too few outcomes to mean anything - which collides directly with the minimum-sample-size requirement from Part 2. Bucket by the dimensions where behaviour genuinely differs, and no further. The temptation to add one more dimension should be met with the question of how many outcomes will land in the resulting cells per week.

Offline Evals and Online Calibration Are Different Tools

These get conflated under the single word "evals," and they answer different questions.

An offline eval runs the system against a fixed set of cases with known-good answers - a golden set - and reports a score. It is repeatable, fast, available before anything ships, and it is the right instrument for a change you are about to make: a prompt edit, a model swap, a restructured specialist. It answers did this change make things better or worse against cases we have already judged.

Online calibration is the loop described above: live predictions scored against real outcomes as they resolve. It is slow, noisy, only available after the fact, and it is the only instrument that tells you how the system is doing now, on the actual distribution of cases, including the ones nobody thought to put in the golden set. It answers is this specialist still reliable.

Neither substitutes for the other. A golden set goes stale, and worse, it is drawn from cases that were interesting when it was built, which is a biased sample of what production sends. Online calibration cannot tell you whether a proposed change is an improvement, because you cannot run a counterfactual against outcomes that already happened. Systems that build only golden sets ship confidently and drift silently. Systems that build only online calibration detect drift but cannot safely change anything, because every change is unmeasurable until enough new outcomes accumulate.

In practice the golden set is usually the piece that gets deferred, built after the system is already working if at all, with mandatory human approval covering the immediate risk in the meantime. That ordering is defensible, but it is worth naming as a gap rather than an absence, because it is the gap that makes prompt and model changes hard to evaluate before they ship, which is exactly the problem Part 3 of the next series runs into with model swaps.

The Outcome Lag Problem

Predictions are cheap and immediate. Outcomes are neither, and the gap between them is where eval harnesses quietly go wrong.

Some outcomes resolve in seconds. Some take days. Some never resolve at all, because nothing observable distinguishes the good case from the bad one, and some resolve only through a human action that may itself be wrong. A harness has to tolerate all of that without either blocking on it or, worse, scoring early against partial information and feeding a confident wrong weight back into the orchestrator.

Three rules that follow:

• Score when the outcome is actually known, not when it is convenient to check. A scheduled job that scores everything from the last hour will systematically mis-score anything whose resolution takes longer than an hour, and the bias is not random - it over-weights fast-resolving cases, which are usually the easy ones.
• Represent "unresolved" explicitly. An outcome that has not arrived yet is not a failure, and collapsing the two is a common and consequential bug. Unresolved cases belong in neither the numerator nor the denominator until they resolve.
• Know the lag distribution per decision type, because it sets a floor on how fast calibration can respond to drift. If outcomes take a week, the system cannot detect a behaviour change faster than a week, and the demotion rules from Part 2 have to be designed knowing that.

Correlated Error and Gaming

Two failure modes attack the integrity of the measurement itself rather than its mechanics.

Correlated error breaks the independence a panel's aggregation assumes. If two specialists are scoring substantially the same signal, their mistakes arrive together, and the orchestrator reads that agreement as corroboration - so the panel is most confident exactly when it is jointly wrong. Individual calibration figures look fine, because each specialist is individually well calibrated; the aggregate is what lies. This is a hard problem and it recurs at a deeper level in the final post of the operations series, where shared infrastructure turns out to create correlation even between specialists that look independent.

Gaming is subtler and does not require intent. Any scored system creates pressure toward whatever the score rewards. A specialist tuned against an accuracy metric can improve its number by abstaining on hard cases, or by hedging toward the base rate, both of which raise the score while reducing the specialist's actual usefulness - the hard calls are the ones the system needed an opinion on. The defence is to score coverage alongside accuracy, so that declining to answer is visible rather than free, and to bucket by difficulty so that easy-case performance cannot mask hard-case abstention.

The Honest Bill

The third line on the right is the one to take seriously. An eval harness wired into production is not auxiliary tooling running alongside the system. It feeds a weight into every decision, which means a bug in the aggregation - a mis-joined outcome, an off-by-one window, an unresolved case counted as a failure - does not produce a wrong dashboard. It produces wrong decisions, with full confidence, and it does so quietly because the thing that would normally catch a wrong decision is the very system now being misinformed. It deserves the same test coverage and the same review discipline as the execution path.

When NOT to Build One

• Outcomes are genuinely unobservable. If nothing distinguishes a good decision from a bad one after the fact, no harness is possible, and the honest response is to keep humans in the loop permanently rather than build machinery that measures nothing. This is a design smell worth addressing at the source: it is often possible to make an outcome observable by capturing a cheap human signal at review time.
• The system does not route on reliability. If nothing consumes the score - no weighting, no autonomy promotion - the harness is a report. Reports are useful, but build the thing that consumes the number before the thing that computes it, or the number will be wrong in ways nobody notices.
• Volume is too low for statistics. A decision type firing twice a week will not accumulate enough outcomes for a meaningful figure within any useful window. Review those individually.
• The specialist is deterministic. If a "specialist" is a threshold check wearing a model's clothes, test it. Reserve calibration for components that exercise judgement.

Conclusion

An eval harness is three things: a recorded prediction, a recorded outcome, and a wire from the aggregate back into the system's behaviour. The first two usually exist already, often for debugging and reporting; the third is what turns them into a harness. Score calibration rather than raw accuracy, because an orchestrator weighing several specialists needs their confidences to be comparable. Bucket until the buckets stop having enough data to mean anything. Keep offline golden sets and online calibration as separate tools answering separate questions, and respect outcome lag as a hard floor on how quickly the system can notice it has drifted.

That closes the decision layer: how to split the work, how to route it, and how to know whether the routing deserves to be trusted. The next series turns from the decision to the action - what an agent is actually permitted to do, what gets recorded about it, and what all of this is running on.