Agentic Systems: What Fan-Out Costs You

This is Part 2 of Agentic Systems: Operating at Scale, the third series in a pillar on building agentic systems.

• Part 1: Memory Is a System, Not a Prompt Trick: the four stores that get collapsed into one word, and what each is for.
• Part 2: What Fan-Out Costs You: the invoice the architecture diagram does not show.
• Part 3: Containing a Bad Specialist: drift, correlated failure, and what containment actually requires.

Part 1 ended by scoping context per specialist to preserve the panel's independence, which means nine specialists assemble nine contexts and make nine calls. This post is about what that actually costs, because the architecture diagram from the first post in this pillar shows arrows and not invoices.

Table of Contents

• The Diagram Hides the Invoice
• Panels Multiply Cost, Pipelines Add Latency
• Lever 1: Model Tier
• Lever 2: Call Frequency
• Lever 3: Cacheability
• The Latency Floor Is the Slowest Specialist
• The Honest Bill
• When NOT to Optimise This
• Conclusion

The Diagram Hides the Invoice

A panel of nine specialists feeding an orchestrator is one box and nine arrows on a whiteboard. In production it is ten billed model calls per decision - nine specialists plus the orchestrator's synthesis - each carrying its own context, each with its own latency distribution and failure probability.

That multiplier is the price of everything the first post argued for. Decomposability, independent evaluability, differential trust, graceful degradation: all of it is bought with a tenfold increase in calls per decision relative to the single-agent version. The benefits accrue over months as calibration accumulates. The cost arrives on the first invoice, which is the asymmetry that makes fan-out feel like a mistake right up until the moment the calibration data starts paying off.

The mistake is not fanning out. It is fanning out uniformly - treating nine specialists as nine instances of the same compute decision, when they differ enormously in what they actually need.

Panels Multiply Cost, Pipelines Add Latency

The panel-versus-pipeline distinction has a precise cost signature, and it is worth having the arithmetic in front of you when choosing a shape.

Assume a specialist call averages 800ms and some unit cost c.

A panel of nine, run concurrently: cost is 9c, latency is roughly 800ms, because they overlap. Cost scales with width; latency does not.

A pipeline of three sequential stages: cost is 3c, latency is 2,400ms, because each stage waits for the one before it. Latency scales with depth; cost does not.

The practical consequence is that the two shapes fail against different budgets. A panel breaks a cost budget long before a latency budget. A pipeline breaks a latency budget long before a cost budget. Knowing which constraint actually binds - and it is usually obvious once asked - tells you which shape you can afford, and sometimes overrides the decomposition the domain would otherwise suggest.

A signal-driven trading panel is affordable partly because its decisions are not interactive. Researchers run continuously against market data on their own cadence rather than nine calls firing on a user's click, so the fan-out cost is a steady background rate rather than a per-request multiplier. That is a genuine architectural advantage of continuous systems over request-driven ones, and it is worth noticing whether a system has it before assuming a panel is too expensive.

Lever 1: Model Tier

The largest available model for every specialist is the default, and it is almost always wrong.

The specialists in a panel are not doing comparable work. One scores whether a metric crossed a threshold, with a short prompt and an enumerable output. Another synthesises ambiguous evidence into a judgement. The first is close to a deterministic computation with a language-model interpretation layer; the second genuinely needs reasoning capability. Paying frontier-model rates for the first buys nothing, because its accuracy ceiling was never set by reasoning depth - it was set by the quality of the threshold.

The trading system's researchers are not one weight class: the heavier reasoning work, sentiment and thematic discovery and synthesis, runs against a cloud model, while narrower scoring work does not need the same treatment. The swappable model layer is what makes this configurable per specialist rather than a code change, which is the practical reason that abstraction earns its keep.

The honest way to set tier is empirical: run the specialist at two tiers against the offline golden set and compare. Frequently the smaller model is indistinguishable on the narrow task, and the difference is pure margin. Occasionally it is much worse, which is also worth knowing before the invoice forces the question.

Lever 2: Call Frequency

Not every specialist needs to run on every decision, and this lever is usually larger than model tier because it removes calls rather than making them cheaper.

Three patterns worth recognising:

Slow-changing signals. A market-regime assessment or a service-maturity classification does not change between two decisions thirty seconds apart. Recomputing it per decision is spending money to re-derive a value that could not have moved.

Conditional relevance. Some specialists only matter for some decisions. An ETF-decomposition specialist has nothing to say about a decision that does not involve one. A gate that skips irrelevant specialists costs a cheap classification and saves a full call, and the saving grows with panel width.

Tiered escalation. Run a cheap subset first; escalate to the full panel only when the cheap subset disagrees or returns low confidence. This is the most effective version of the lever, because it concentrates spending on genuinely hard decisions - which are usually a minority - and it makes the cost profile track difficulty rather than volume.

Escalation has a real caveat: it changes what the calibration data means. A specialist that only runs on hard cases has a track record drawn from hard cases, so its accuracy is not comparable to one that runs on everything. That needs to be reflected in the bucketing, or the escalation-only specialists will look worse than they are and get wrongly demoted.

Lever 3: Cacheability

If a specialist's inputs have not changed, its output has not changed either, and recomputing is waste.

The discipline is that a cache must be keyed on exactly what would invalidate it. A specialist reading an alert payload, a service map, and a recent-incident window is cacheable on the hash of those three - and a cache keyed on anything less than the full input set will serve stale answers silently, which is worse than being slow, because a stale specialist signal looks current to the orchestrator and gets weighted as if it were.

Two further notes. Provider-side prompt caching is a different and complementary mechanism: it reduces the cost of re-sending a large shared prefix rather than avoiding the call, and whether it is available is one of the capability differences a model abstraction must not hide. And caching interacts with the ledger - a cached decision still needs a ledger entry, or the record will show fewer decisions than the system actually made, which corrupts every calibration figure computed from it.

The Latency Floor Is the Slowest Specialist

Concurrency bounds a panel's latency at its slowest member, not its average, and this is where fan-out budgets are most often wrong.

With nine concurrent calls, the decision completes when the last one returns. If eight average 600ms and one averages 2s, the panel's latency is 2s. Worse, tail latency dominates: the probability that at least one of nine calls lands in its own slow tail is far higher than the probability that any particular one does, so the panel's typical latency tracks the specialists' p95 rather than their median.

Three mitigations, each with a cost:

• Timeout and proceed without the stragglers. The orchestrator aggregates whatever arrived. Cheap, and it means some decisions are made on a partial panel - which must be recorded in the ledger, or calibration will score specialists on decisions they did not participate in.
• Move slow specialists off the critical path. If a specialist's signal is slow-changing, compute it asynchronously and read the last value, converting a latency cost into a staleness cost.
• Reduce the slow specialist's work. Usually its context is large. Scoping context per specialist, as Part 1 argued for independence reasons, turns out to help here too.

The Honest Bill

The calibration-skew line is the one that catches teams by surprise, because the cost optimisation and the measurement system are usually owned by different concerns and changed at different times. Any lever that changes which decisions a specialist sees changes what its track record means. Cost tuning is not behaviour-neutral, and it should go through the same evaluation as a model swap.

When NOT to Optimise This

• Before there is traffic. Optimising fan-out on a system making forty decisions a day is effort spent on an invoice that does not exist. Build it uniformly, measure, then tune.
• When the panel is small. Three specialists is a 3x multiplier. The configuration complexity of per-specialist policy may exceed the saving.
• When cost is not the binding constraint. If latency or accuracy is what limits the system, cost tuning is the wrong axis and may make the real constraint worse.
• When the cheaper model measurably degrades the decision. Run the comparison. If the smaller model is worse on the golden set, the saving is being taken out of decision quality, which is a different trade than the one being contemplated.
• When the specialist is already off the critical path. Asynchronous, slow-changing specialists are usually a rounding error. Optimise what actually fires per decision.

Conclusion

A panel multiplies cost by its width and floors latency at its slowest member; a pipeline adds latency by its depth and compounds error along it. Neither is free, and the bill arrives long before the calibration benefits do. Three levers control it: model tier matched to what a specialist actually requires, call frequency reduced by skipping irrelevant specialists and escalating only on hard cases, and caching keyed on exactly the inputs that would invalidate it. Each one changes which decisions a specialist sees, which means each one is also a change to the measurement system and deserves the same evaluation discipline.

Tuning assumes the specialists are working. Part 3 takes the case where one of them is not, and finds that the independence a panel relies on is only as real as the independence of what it is all running on.