Scaling: What Breaks, and When

This is Part 2 of Scaling: Orders of Magnitude, the first series in a pillar on scaling systems.

• Part 1: Reading the Numbers: what a throughput target actually means, and why the raw number lies.
• Part 2: What Breaks, and When: walking one system up the orders of magnitude and naming what cracks first at each rung.
• Part 3: The Menu of Levers: the survey of everything you can reach for, on both the API and the client side.
• Part 4: Choosing Under Constraints: how to actually decide, and how the domain changes the dials.

Part 1 was about reading the numbers honestly. Part 2 uses those numbers to answer the question every scaling conversation is really asking: at the load we actually have, what is going to break first, and what is the cheapest thing that fixes it? We answer it by taking the food-delivery app from Part 1 and walking it up the orders of magnitude, one rung at a time.

Table of Contents

• The Ladder
• Hundreds of Requests per Second: One Box
• Thousands: Caching and Read Replicas
• Tens of Thousands: Statelessness and Horizontal Scale
• Hundreds of Thousands: The Sync Path Cracks
• Millions: Physics Enters the Design
• The Thread Running Through It: State Is What Hurts
• Conclusion

The Ladder

Scaling is not a single event. It is a sequence of ceilings. You hit one, you raise it, you run for a while, you hit the next. The mistake that wastes the most effort is trying to raise a ceiling you have not hit yet, building the sharding scheme you will need at 100,000 requests per second while you are serving 800.

Each rung of the ladder has a characteristic thing that breaks first and a characteristic cheapest fix. Knowing the ladder means you can look at your numbers from Part 1, locate yourself on it, and reach for the right fix instead of the impressive one. Here is the whole ladder in one table before we walk it.

Notice that the fixes are cumulative. You do not throw away caching when you add sharding. Each rung keeps everything below it and adds one more idea.

Hundreds of Requests per Second: One Box

At a few hundred requests per second, the correct architecture is boring, and boring is correct.

One application server and one database will handle it comfortably. A modern database on decent hardware serves thousands of simple queries per second without complaint. At this scale your bottleneck is not throughput, it is you: the speed at which you can ship features, find product-market fit, and not drown in operational complexity you do not need.

The failure mode at this rung is not falling over. It is over-engineering. A team that reads about Kafka and sharding and multi-region failover, and builds all of it to serve 300 requests per second, has spent its scarcest resource, engineering time, buying capacity it will not use for years and may never use at all. Part 4 has more to say about premature optimization as the default failure mode. For now: at hundreds of requests per second, resist. One box, one database, good instrumentation so you can see the next ceiling coming. That is the whole design.

Thousands: Caching and Read Replicas

Somewhere in the low thousands of requests per second, the single database starts to feel it, and because the food app is read-heavy, it feels it on reads first.

The same restaurant list, the same menus, the same popular dishes are being read thousands of times per second, and each read is doing real work against the database when almost none of it changes minute to minute. This is exactly what caching is for. Put a cache in front of the hot reads and the database load for those paths collapses, because you have stopped asking it the same question thousands of times.

The two cheapest, highest-return moves at this rung are:

• A read cache for the hot, slow-changing reads: restaurant lists, menus, static catalog data. This is the single highest-leverage change most systems ever make, because it attacks the read/write asymmetry from Part 1 directly. Reads want copies, and a cache is a copy.
• Read replicas for the reads that cannot be cached but can tolerate slight staleness: order history, past receipts. Replicas let you add read capacity by adding copies of the data.

Both work because reads do not have to agree with each other in real time. Neither does anything for writes, which still go to the one primary database. That is fine here, because at thousands of requests per second and a 95/5 read/write split, the write load is still a few hundred writes per second, which one database handles easily. The write ceiling is real, but it is not this rung's problem.

Tens of Thousands: Statelessness and Horizontal Scale

In the tens of thousands of requests per second, one application server is no longer enough, and how you scale out depends entirely on a decision you hopefully made much earlier: whether your services hold state.

If each request can be handled by any server interchangeably, scaling out is trivial. Put a load balancer in front, run twenty servers instead of one, add more when the peak grows. This is horizontal scaling, and it is close to free precisely because the servers are stateless. They compute, they read and write the shared datastore, they keep nothing local that another request needs.

If, on the other hand, your servers hold session state in memory, so that a user's requests must return to the same server, then scaling out is a fight. You need sticky sessions, and sticky sessions break autoscaling, complicate deploys, and fall over when a server dies and takes its users' state with it. The cheapest fix at this rung is therefore mostly a fix you make before you get here: keep application servers stateless, push session state into a shared store or a token the client carries, and let any server serve any request.

Two other things start to bind at this scale and are worth naming:

• Connection pools. Every app server holds a pool of connections to the database. Twenty servers with fat pools can open more connections than the database will tolerate, and you discover this at peak, in the worst way. Pool sizing becomes a real constraint.
• The load balancer and its own limits. It is now a component with capacity and failure modes of its own, not an invisible arrow on a diagram.

The database, meanwhile, is still a single primary, and the writes are climbing. We are now within sight of the ceiling this whole pillar is really about.

Hundreds of Thousands: The Sync Path Cracks

At hundreds of thousands of requests per second, two things break at once, and they are the two things the rest of the pillar exists to address.

The single database can no longer hold the writes. You have cached the reads and replicated them, but every replica descends from one primary, and that primary is one machine with one disk. When the write rate exceeds what one machine can durably commit, no amount of read scaling helps. The fix is to stop having one database. You shard: split the data by some key, the customer or the region or the restaurant, so that different writes go to different databases and the write load spreads across many machines. Sharding is powerful and it is expensive in complexity: cross-shard queries get hard, transactions across shards get very hard, and choosing the shard key is a decision you will live with for years. It is the price of scaling state, and it is why state is the hard part.

The synchronous request-response path cracks under its own coupling. When placing an order means synchronously calling seven downstream services, the whole chain is only as available as its least available link and only as fast as its slowest one. At high volume, a slow payment gateway or a struggling notification service does not just slow itself down, it holds open every order request waiting on it, exhausts the connection pool, and takes down the front door. The fix is to stop doing everything synchronously. The things that do not have to happen before you answer the customer, notifying the restaurant, assigning a courier, sending receipts, get moved off the critical path into an asynchronous flow behind a queue or a log. The customer gets a fast confirmation; the rest happens durably in the background. This is the exact point where a broker like Kafka earns its place, and the API series is devoted to doing it well.

This rung is the heart of the pillar. Sharding is how you scale state; asynchronous decoupling is how you scale the flow. Almost everything in the API and UI series is a detailed treatment of one or the other.

Millions: Physics Enters the Design

At millions of requests per second, you are usually not in one place anymore, and once you span the globe, physics stops being an abstraction and starts being a design input.

A single region has limits: power, capacity, and the simple fact that a data center in Virginia serves a customer in Singapore across 200-plus milliseconds of round trip that no engineering can remove, because the speed of light is fixed. To serve a global audience at low latency you run in multiple regions, close to users. And the moment you have more than one region, you inherit the hardest problem in the field: keeping data consistent across places that cannot talk to each other instantly. Coordination costs latency, and at global distances that latency is large enough that you often cannot afford strong consistency across regions at all. You are forced to choose, per piece of data, between consistency and availability. That is not a tuning knob; it is the CAP theorem showing up in your architecture, and Part 4 gives it the vocabulary it deserves.

I am naming this rung, not dwelling on it. Multi-region is deliberately out of scope for the deep dives in this pillar, because most systems never need it and the ones that do need a book, not a section. The point for now is the shape of the lesson: at the top of the ladder, the constraints stop being about your code and start being about the universe your code runs in.

The Thread Running Through It: State Is What Hurts

Step back from the rungs and one thread runs through all of them.

At every rung, the thing that broke was state. The stateless application tier scaled almost for free the entire way up: add boxes, add a load balancer, done. Every genuinely hard problem was a state problem. Caching is about keeping copies of state. Replicas are copies of state. Sharding is splitting state. Async decoupling is about not blocking on state changes you can defer. Multi-region is about state that cannot agree with itself across distance.

This is the single most useful lesson in the series, so it is worth stating plainly: most of what we call scaling is really scaling state. Computation scales by adding machines. State resists, because state has to be consistent, durable, and in the right place, and those three demands fight both each other and the physics of distance. When you look at a system and ask what will break first, the answer is almost always found at the data layer. Design accordingly, and you will spend your effort where the difficulty actually is.

Conclusion

Scaling is a ladder of ceilings, and each rung has a characteristic thing that breaks first: nothing at hundreds, read load at thousands, single-server and session limits at tens of thousands, the single database and the synchronous path at hundreds of thousands, and physics itself at millions. The fixes are cumulative, and the cheapest one is almost always the least impressive. Locate yourself on the ladder using the numbers from Part 1, and reach for the fix that rung actually calls for.

Running under all of it is one truth: state is what hurts. Knowing that tells you where to look. Part 3 lays out the full menu of levers, on both the API and the client side, that you reach for as you climb, so that when you hit a ceiling you know your options rather than reinventing them under pressure.