- Published on
Scaling: The Menu of Levers
- Authors
This is Part 3 of Scaling: Orders of Magnitude, the first series in a pillar on scaling systems.
- Part 1: Reading the Numbers: what a throughput target actually means, and why the raw number lies.
- Part 2: What Breaks, and When: walking one system up the orders of magnitude and naming what cracks first at each rung.
- Part 3: The Menu of Levers: the survey of everything you can reach for, on both the API and the client side.
- Part 4: Choosing Under Constraints: how to actually decide, and how the domain changes the dials.
Part 2 walked the ladder of ceilings. Each time you hit one, you reach for a lever. Part 3 lays out the whole rack of levers in one place, on both sides of the wire, so that when you are under pressure you are choosing from a known menu instead of inventing options in the moment. This is a map, not a manual: each lever gets what it buys and what it costs, and the API and UI series are where we actually pull them.
Table of Contents
Two Sides of the Wire
There are two places you can do work: on your servers and on the client's device. Most scaling discussions focus entirely on the first, which is a mistake, because the cheapest request your backend ever handles is the one it never receives. A byte served from a CDN edge, or an answer the client already had cached, never touched your origin at all.
So the menu has two columns. The API-side levers make your servers handle more. The client-side levers make your servers handle less, by pushing work and data outward toward the user. A well-scaled system pulls levers from both columns, and the two are deeply connected: an API decision to go asynchronous, for example, forces a client decision about how to show work that is not done yet. Keeping both columns in view is the whole reason this pillar has a backend series and a frontend series rather than just the first.
API-Side Levers
Horizontal Scale and Statelessness
What it buys: near-linear capacity for the compute tier. Run more identical servers behind a load balancer and handle proportionally more requests.
What it costs: almost nothing, provided your services are stateless. The cost is paid earlier, in the discipline of not keeping request-critical state in server memory. If you skipped that discipline, this lever jams: sticky sessions, painful deploys, state lost when a server dies. Statelessness is the enabling condition for most of the other levers, which is why Part 2 kept returning to it.
Caching
What it buys: the largest single reduction in load most systems ever achieve. Serving a hot read from memory instead of computing it against the database collapses the load on the expensive path. It attacks the read/write asymmetry directly, because reads want copies and a cache is a copy.
What it costs: correctness worry, in one word, invalidation. A cache is a second copy of the truth, and the moment the truth changes the copy is wrong. How stale can you tolerate, how do you invalidate, what happens when the cache is cold or falls over: these are the real questions, and they are not trivial. But the payoff is so large that caching is usually the first lever to reach for.
Read Replicas and Sharding
What it buys: database scale. Replicas add read capacity by copying the data; sharding adds write capacity by splitting the data across independent databases. Together they are how you scale the data layer, which Part 2 identified as the real ceiling.
What it costs: replicas cost you replication lag, a read from a replica may not reflect the most recent write. Sharding costs you a great deal more: cross-shard queries and transactions get hard, the shard key is a near-permanent commitment, and rebalancing is painful. Sharding is powerful and it is the point where operational complexity rises sharply, so it is a lever you pull when you must, not when you can.
Load Balancing
What it buys: distribution of traffic across your fleet, plus the health checking and failover that let a dead server drop out without taking requests with it. It is the component that makes horizontal scale usable.
What it costs: it becomes a tier of its own, with capacity, configuration, and failure modes to manage. Choices like sticky versus stateless routing, and how health is judged, have real consequences, especially once you are holding long-lived connections, which the UI series returns to.
Asynchronous Processing and a Broker
What it buys: the deepest lever in the pillar. Moving work off the synchronous critical path into a queue or a log (a broker such as Kafka) decouples producers from consumers, absorbs spikes by buffering instead of dropping, lets one event fan out to many independent consumers, and gives you durability and replay. It is how you break the coupling that makes the synchronous path crack at high volume.
What it costs: genuine complexity. You trade immediate consistency for eventual consistency, add end-to-end latency for the deferred work, take on new operational burden, and make the system harder to debug because the flow is no longer a straight line you can follow. This lever is powerful enough, and costly enough, to justify an entire series. The API series is largely about pulling it well, and about the many cases where you should not pull it at all.
Batching
What it buys: throughput, by amortising fixed per-operation costs. Writing a thousand records in one batch is far cheaper than a thousand single writes, because the overhead per item shrinks. The same idea covers batched reads and coalesced downstream calls.
What it costs: latency and complexity. Batching means waiting to accumulate a batch, which delays individual items, and it complicates error handling, because now a partial failure inside a batch has to be reasoned about. It trades a little latency for a lot of throughput, which is exactly the right trade for background work and the wrong one for the critical path.
Backpressure and Rate Limiting
What it buys: survival under overload. When demand exceeds capacity, a system without backpressure does not politely slow down, it collapses, because queues grow without bound until something runs out of memory. Backpressure and rate limiting let the system shed or slow load deliberately, degrading instead of dying.
What it costs: rejected or delayed requests, and the product conversations about who gets throttled and how it is communicated. It is the difference between a bad evening and an outage, so the cost is almost always worth paying, but it has to be designed in, not bolted on after the first collapse.
Client-Side Levers
The CDN and the Edge
What it buys: the cheapest request is the one that never reaches you. A content delivery network serves cached responses and static assets from an edge location near the user, cutting both origin load and latency. For read-heavy, cacheable content, images, static pages, menus that change slowly, it removes enormous load from your origin entirely.
What it costs: cache invalidation again, now spread across a global network of edges, plus the reality that dynamic and personalised content is harder to push to the edge. It shines for the static and slow-changing, and needs care for anything user-specific.
Client Caching
What it buys: load you never see and latency the user never feels, by keeping data on the device and reusing it. HTTP caching, application-level caches, and on-device storage all mean the client answers its own question instead of asking you again.
What it costs: staleness on the device, and the logic to decide when the client must revalidate versus trust what it holds. For native apps this extends to on-device databases as a real cache tier, which the UI series covers.
Payload Shape and Pagination
What it buys: less data over the wire and less work on both ends. Returning only the fields the screen needs, paginating long lists instead of shipping ten thousand rows, and avoiding client-side N+1 request storms all cut bandwidth, server load, and render time at once.
What it costs: API design effort and discipline. Right-sized payloads mean the API has to be shaped around real screens, and pagination pushes complexity, cursors, ordering, stability, onto both client and server.
Real-Time Transports
What it buys: live updates. When the customer needs to watch the courier move or see the order status change, the choice of transport, polling, long-polling, server-sent events, or websockets, determines both freshness and cost.
What it costs: connection load, and it can be severe. Holding a hundred thousand concurrent open connections is itself a scaling problem, in memory, in load balancers, and in fanning updates out from the backend. The transports trade simplicity against freshness against connection cost, and the UI series works through the trade in detail.
Optimistic UI
What it buys: perceived speed, and a graceful answer to an asynchronous backend. Instead of waiting for the server to confirm, the client updates immediately as if the action succeeded, then reconciles when the real answer arrives. The user feels an instant response even though the work is still in flight.
What it costs: you might be wrong. If the action ultimately fails, the client has to walk the user back from a success it already showed, which is delicate. And optimism is domain-sensitive: a food app can optimistically show an order placed, while a bank refuses to show a transfer as done before it truly is. The UI series treats this as the point where the whole pillar pays off on the client side.
Every Lever Has a Bill
The single most important thing to carry out of this menu is that there are no free levers. Every one buys capacity or speed with a cost paid in consistency, complexity, latency, or money.
| Lever | Buys | Pays in |
|---|---|---|
| Horizontal scale | Compute capacity | Requires statelessness |
| Caching | Massive read load reduction | Invalidation, staleness |
| Replicas / sharding | Data-layer scale | Lag; deep operational complexity |
| Async / broker | Decoupling, spike absorption | Eventual consistency, debugging |
| Batching | Throughput | Latency, error-handling |
| Backpressure | Survival under overload | Rejected or delayed requests |
| CDN / edge | Origin offload, low latency | Invalidation; dynamic is harder |
| Client caching | Requests you never see | Device staleness |
| Real-time transports | Live updates | Connection load at scale |
| Optimistic UI | Perceived speed | Being wrong; domain sensitivity |
Read the table as a list of exchange rates. Scaling well is not about pulling every lever; it is about pulling the few whose bill you are willing to pay for the capacity you actually need. Which bills you are willing to pay is decided by your constraints and your domain, which is exactly where Part 4 goes.
Conclusion
The menu has two columns because there are two places to do work, your servers and the client's device, and a well-scaled system reaches into both. The API-side levers make servers handle more; the client-side levers make servers handle less. Every lever buys something real and charges for it in consistency, complexity, latency, or cost, and none is free.
This part was the map. Part 4 is about reading the map under real constraints: how you actually decide which levers to pull, and how the domain you are in, banking versus commerce versus advertising, quietly changes every answer by changing which cost you refuse to pay.