Agentic Systems: Autonomy You Have to Earn

This is Part 2 of Agentic Systems: The Decision Layer, the first series in a pillar on building agentic systems.

• Part 1: Orchestrators and Specialists: how many agents, what each owns, and the contract between them.
• Part 2: Autonomy You Have to Earn: routing a decision to a human or to execution, and why that gate has to be earned per decision.
• Part 3: The Eval Harness You Already Have: scoring specialists against outcomes, and feeding that score back into the system.

Part 1 ended with a decision routed by an orchestrator. This post is about where it gets routed. That choice is usually made once, globally, at design time, and then never revisited, which is how a system ends up either drowning its operators in approvals or acting unsupervised on decisions it was never reliable at.

Table of Contents

• Two Defaults, Both Wrong
• Autonomy Is a Property of Decisions, Not Systems
• What "Earned" Has to Mean in Numbers
• The Third Axis Nobody Budgets For: Rate
• The Demotion Path Is the Harder Half
• The Veto Window
• Decisions That Should Never Earn It
• The Honest Bill
• When NOT to Automate at All
• Conclusion

Two Defaults, Both Wrong

Return to the alert-triage system from Part 1. A decision has been routed: severity assessed, deploy correlation checked, ownership attributed, and a paging recommendation produced. Now what happens?

The first default is that a human approves everything. Every recommendation lands in a queue and waits. This is the responsible-sounding choice, and it fails in a specific way: it does not survive volume. At ten decisions a day it is fine. At four hundred, the operator is no longer evaluating recommendations, they are clearing a backlog, and the approval step has quietly become a rubber stamp that provides the appearance of oversight without the substance. Worse, the failure is invisible. The metrics look identical whether a human is reading carefully or clicking through.

The second default is that the system acts alone. This fails because it assumes the system's reliability is uniform across every decision it might make, which is never true. The triage system might be excellent at recognising a database connection-pool exhaustion it has seen a thousand times, and poor at classifying a novel failure in a service deployed last week. One global autonomy setting cannot express that difference. It has to be set for the worst case, which wastes the good case, or for the average, which quietly automates the bad case.

"Human in the loop," as the phrase is usually deployed, is just the first default with better branding. The interesting question was never whether a human is in the loop. It is which specific decisions have earned the right to skip it.

Autonomy Is a Property of Decisions, Not Systems

The reframe that makes this tractable is to stop treating autonomy as a system-level configuration and start treating it as an attribute of a decision type - a named, recurring category of call that the system makes, narrow enough that its outcomes can be pooled and counted.

"Page the owning team for a connection-pool exhaustion on a service with a known runbook" is a decision type. "Handle incidents" is not; it is a job description. The test is whether you could, at the end of a quarter, produce a list of every instance of it and say how many turned out well. If you cannot enumerate and score it, it cannot earn anything, because there is no record for it to earn against.

This is where Part 1's argument pays off. A single agent doing four competencies has exactly one reliability number, so it has exactly one autonomy setting available: all or nothing. Decomposing into specialists creates units whose reliability means something specific, and that specificity is what makes differential trust possible. Earned autonomy is not a separate feature you add later. It is the thing the decomposition was for.

What "Earned" Has to Mean in Numbers

"Proves reliable over time" is not a specification. Before any decision type is promoted, four things need concrete values.

A minimum sample size. This is the one most often skipped, and the arithmetic is unforgiving. Suppose a decision type has produced twenty consecutive good outcomes, which feels like strong evidence. For a process that is truly 95% reliable, the probability of twenty consecutive successes is 0.95^20, about 36%. For one that is truly 99% reliable, it is 0.99^20, about 82%. Both are entirely ordinary outcomes. A twenty-success streak is consistent with a system that fails one time in twenty and with one that fails one time in a hundred, and those are very different systems if the decision is expensive.

Read the chart at a streak of twenty. A 95%-reliable process produces one about a third of the time, so seeing one tells you very little about whether you are looking at 95% or 99%. The curves only separate meaningfully once the streak runs long, which is another way of saying that short streaks are not evidence, they are the absence of counter-evidence. The two get confused constantly, and the confusion always points toward promoting too early.

An accuracy threshold, set against the cost of being wrong. Ninety-five percent is excellent for classifying a support ticket and unacceptable for a decision that pages someone at 3am, because the 5% is not a rounding error, it is a person woken for nothing, repeatedly, until they stop trusting the pager. The threshold is a product decision expressed as a number, not a statistical convention.

A window. Reliability measured over all of history weights a year-old track record equally with last week's, which is exactly wrong when the environment moves. Reliability measured over the last ten decisions is noise. Somewhere in between is a window that reflects current behaviour without amplifying variance.

A bucketing rule. Accuracy pooled across dissimilar cases hides the distribution, which is the same failure Part 1 identified in composite agents, one level up. A decision type reliable on services with runbooks and unreliable on services without them has a pooled number that describes neither.

Write these four down before building the promotion machinery. If they cannot be stated, the system does not have earned autonomy, it has a switch someone will eventually flip on a hunch.

The Third Axis Nobody Budgets For: Rate

The familiar frame for deciding what needs supervision is blast radius against reversibility, and an earlier series on this blog worked through that matrix in the context of supervising coding agents. It holds here. It is also incomplete for systems that decide continuously.

A single decision with small blast radius and easy reversal is safe. The same decision made four hundred times a day is a different object. Individually trivial actions aggregate: four hundred slightly-wrong notifications is an operator who has stopped reading them, and four hundred small position adjustments is a portfolio that has drifted somewhere nobody chose. Reversibility per instance also does not imply reversibility in aggregate, because undoing four hundred interleaved actions is not four hundred times as easy as undoing one.

So the third axis is rate. A decision type's autonomy case should be argued on blast radius, reversibility, and how often it fires. High-frequency decisions deserve tighter thresholds than their per-instance blast radius suggests, precisely because their per-instance blast radius is not what you will eventually be dealing with.

The Demotion Path Is the Harder Half

Promotion gets the attention. Demotion is what makes promotion safe, and it is the part most systems never build.

If a promoted decision type starts failing more than its track record predicted, it needs to return to requiring approval - automatically, on the evidence, without waiting for somebody to notice a pattern and intervene. A system where autonomy only ever increases is not a calibration system. It is a ratchet with a calibration story attached.

The useful discipline: for every promotion rule, write the inverse before shipping it. If promotion is "at least N outcomes in the window, at or above threshold T," demotion is what happens at T minus some margin, over what window, with what hysteresis so a decision type does not oscillate between states on every new data point. If you cannot state what would cause a decision type to lose its automation, you have not defined earned autonomy. You have defined a one-way switch.

Demotion also needs to be cheap to trigger and slightly expensive to undo, rather than symmetric. The cost of wrongly demoting is some extra human review. The cost of wrongly retaining autonomy is whatever the decision does when it is wrong. Those are not the same, and the thresholds should not pretend they are.

The Veto Window

Between "propose and wait" and "execute immediately" there is a third state that is underused: execute, but after a visible delay during which the action can be stopped.

The trading system from Part 1 uses this. Every trade begins as manual. As specific signal combinations prove reliable, those combinations earn automatic execution - and even then the trade passes through a veto window first, a short interval where the pending action is visible and can be cancelled before it becomes irreversible. Nothing skips the queue by default; automation is earned per signal combination rather than granted to the system.

The veto window is valuable because it decouples two things that approval queues conflate: reviewing every decision and being able to stop a bad one. A queue requires a human to act for anything to happen, so throughput is bounded by attention. A veto window requires a human to act only to prevent something, so the default path is fast, and attention is spent only where something looks wrong. It converts supervision from a gate into a circuit breaker.

It is not free. It adds latency equal to the window on every automated action, which rules it out where the decision must be immediate, and it only works if someone is actually watching during the window - an unwatched veto window is a delay with a governance story attached.

Decisions That Should Never Earn It

Some decisions should stay supervised no matter how good the track record becomes, and recognising them is part of designing the promotion path rather than an exception to it.

The tender-response system from Part 1 takes this position deliberately: every agent output arrives as a proposal with its reasoning and source clause attached, and nothing enters a submission without a person approving it. That is not a temporary limitation pending better models. When a submission carries a legal representation to a public body, the liability sits with a named person, and no accuracy figure transfers it to software. A decision type whose cost of error is categorical rather than statistical cannot be reasoned about with thresholds, because the threshold framing assumes errors are fungible and survivable.

The general test: if being wrong once is not compensated by being right a hundred times, accuracy-based promotion is the wrong instrument regardless of the numbers.

The Honest Bill

Earned autonomy is real machinery, and it is worth being clear about what it costs before adopting it.

Two lines deserve emphasis. The dependency on Part 3 is total: promotion thresholds are meaningless without a system that records outcomes and computes reliability honestly, so building promotion first produces confident automation resting on nothing.

And over-trust is the failure this whole apparatus is most vulnerable to, because it is a failure of the humans rather than the system. A decision type performs well for months, the reviews become perfunctory, and the veto window goes unwatched. The calibration did not fail. The supervision did, silently, while every dashboard stayed green.

This is not a new observation, and it is worth knowing that the field that studied it first is not software. Lisanne Bainbridge's 1983 paper Ironies of Automation, written about industrial process control, argued that automating the routine parts of a task leaves the operator with the residue: rare, high-stakes interventions, to be performed with skills that have decayed precisely because automation removed the daily practice that maintained them. The irony in the title is that the better the automation gets, the more critical and the less rehearsed the human's remaining role becomes.

That maps onto earned autonomy almost directly. Promoting the decision types a system handles well leaves operators reviewing only the unusual ones, less often, with less recent context - which is the moment their judgement matters most. The mitigation is structural rather than motivational: keep the outcome record visible, keep some sampled review of automated decisions rather than only exceptions, and treat "it has been fine for a while" as a claim requiring evidence rather than a reason to stop looking.

When NOT to Automate at All

• The outcome is not observable. If you cannot tell, after the fact, whether a decision was good, nothing can be earned, because there is no evidence to accumulate. Build the outcome capture first, or keep the human.
• Volume is low enough that review is cheap. At twenty decisions a day, a human reviewing each one is not a bottleneck, and the promotion machinery costs more than it saves. Automate when attention is genuinely scarce, not because automation is available.
• The error is categorical, not statistical. Legal representations, irreversible financial commitments, anything where one bad instance is not offset by many good ones.
• The decision type is too new to have a track record. Every decision type starts unproven. The correct setting for an unproven type is manual, and the temptation to skip that phase because a similar type is reliable is exactly how correlated assumptions get baked in.
• Nobody will watch the veto window. If the answer to "who is looking during those thirty seconds?" is nobody, the window is not supervision, and the decision should either go to a real queue or execute immediately with the risk acknowledged.

Conclusion

Autonomy set once and globally is set wrong, because reliability is not uniform across decisions. The alternative is to name decision types narrow enough to score, define promotion in concrete numbers - sample size, threshold, window, bucketing - and build the demotion path before the promotion path, so trust can be withdrawn as automatically as it was granted. A veto window sits usefully between proposing and executing, converting supervision from a throughput bottleneck into a circuit breaker. And some decisions stay supervised permanently, because their errors are categorical rather than statistical.

All of it rests on one assumption: that something is honestly measuring whether decisions turned out well. Part 3 is about that measurement - what it takes to score a non-deterministic system against reality, why calibration matters more than raw accuracy, and why most teams already have most of the pieces without recognising them as an eval harness.

Sources and further reading:

• Lisanne Bainbridge, "Ironies of Automation", Automatica 19(6), 1983 - the foundational argument that automating routine work leaves operators with rarer, harder interventions and less practice for them.
• Part 3 of "AI State of Play" - the blast-radius and reversibility matrix this post extends with a rate axis.