Scaling: Reading the Numbers

This is Part 1 of Scaling: Orders of Magnitude, the first series in a larger pillar on scaling systems. The pillar is aimed at engineers who scale real systems in production, not at interview prep. It has three series:

• Orders of Magnitude (this series): the mental model. What breaks at each scale, the menu of levers, and how to choose.
• The API / Backend Track: how far synchronous request-response actually goes, where async and a broker like Kafka earn their place, and correctness in an async world.
• The UI / Frontend Track: serving clients at scale, real-time transports, and how the async backend leaks into the user experience.

The four parts of this series:

• Part 1: Reading the Numbers: what a throughput target actually means, and why the raw number lies.
• Part 2: What Breaks, and When: walking one system up the orders of magnitude and naming what cracks first at each rung.
• Part 3: The Menu of Levers: the survey of everything you can reach for, on both the API and the client side.
• Part 4: Choosing Under Constraints: how to actually decide, and how the domain changes the dials.

Throughout the pillar, one system runs as the worked example: a food-delivery app with a payment path. It has a synchronous side (place an order, take payment, see it confirmed) and an asynchronous side (the restaurant accepts, a courier is assigned, status updates get pushed to the customer). That mix is what makes it a good teacher.

Most scaling conversations start with a number. "We need to handle 20,000 requests per second." That number is where the trouble begins, because on its own it says almost nothing about what you actually have to build. Part 1 is about learning to read the number before you design against it.

Table of Contents

• The Number on Its Own Lies
• Peak Versus Sustained
• The Read/Write Mix
• Fan-Out: One Click Is Not One Request
• Throughput and Latency Are Different Problems
• Averages Hide the Failures That Matter
• Know Your Numbers Before You Design
• Conclusion

The Number on Its Own Lies

"20,000 requests per second" feels like a specification. It is not. It is a headline, and like most headlines it hides everything that determines the work.

Twenty thousand reads per second against a cache is a laptop. Twenty thousand payment writes per second, each of which has to be durable, ordered, and reconciled against a ledger, is a serious engineering effort with a serious bill. The two share a number and share nothing else. Before that number can guide a single decision, it has to be decomposed into the properties that actually shape a system: when the load arrives, what kind of work each request does, how many downstream calls one request spawns, and how tightly the latency is bounded.

The rest of this post is those properties, one at a time.

Peak Versus Sustained

The first question to ask about any throughput number is when it happens.

A food-delivery service does not receive a flat 20,000 requests per second all day. It receives a trickle at 4pm, a wall at 7:30pm, and near-silence at 4am. The daily average might be 3,000 requests per second while the dinner peak touches 20,000, and a Friday or a promotion might double the peak again. If you design for the average you fall over every evening. If you design for the all-time peak you pay for idle capacity twenty hours a day.

So a single number is never enough. You need at least three:

The ratio between peak and average is often the most informative single figure in the whole conversation. A system with a peak-to-average of 1.5 can be over-provisioned cheaply and left alone. A system with a peak-to-average of 10, which is entirely normal for anything tied to human meal times or paydays or live events, has to be elastic, and elasticity is a design constraint that reaches into everything from how you cache to whether you can hold connections open.

The Read/Write Mix

The second question is what the requests do. Reads and writes scale by completely different rules.

Reads can be served from replicas, from caches, from a CDN, from the client's own memory. You can add read capacity almost linearly by adding copies of the data, because copies of the same data do not have to agree with each other in real time. A read that is a few hundred milliseconds stale is usually fine.

Writes have to agree. A write has to land somewhere authoritative, be made durable, and become visible to future reads in some defined order. You cannot serve a write from a cache, and you cannot make writes faster by adding read replicas. Write scaling is the hard scaling, and it is why the data layer, not the service layer, is almost always the real ceiling. We will come back to that idea in every part of this pillar.

In the food-delivery example the mix is lopsided and revealing. Browsing restaurants and menus, checking an order's status, watching the courier move on a map, all reads, and the overwhelming majority of traffic. Placing an order and charging a card, writes, and comparatively rare, but they are the writes the entire business depends on. A system that is 95 percent reads and 5 percent writes is not a "20,000 rps system." It is a 19,000 rps read system bolted to a 1,000 rps write system, and those two halves want almost nothing in common.

Fan-Out: One Click Is Not One Request

The third question, and the one most often skipped, is how many requests one request really is.

When a customer taps "Place Order," the button is one action. Behind it, the system might validate the cart, check the restaurant is open, reserve inventory, authorise the payment, create the order record, notify the restaurant, enqueue a courier-assignment job, and write several audit and analytics events. One user-visible request has fanned out into a dozen internal calls, each hitting a different service or datastore, each with its own capacity and its own failure modes.

This is why "requests per second" has to be qualified by where you are measuring. Twenty thousand orders per second at the front door can be two hundred thousand internal calls per second in the middle of the system. The number that sizes your fleet is the internal one, and the only way to know it is to trace a single action all the way through and count.

A useful habit is to draw the fan-out for your most important action explicitly:

One click, seven-plus downstream operations, a mix of reads and writes across five or six systems. Multiply by peak order rate and you have the real load. Skip this exercise and you will size the front door correctly and still fall over in the middle.

Throughput and Latency Are Different Problems

Throughput is how many requests you can handle per unit time. Latency is how long any one request takes. They are related but they are not the same problem, and conflating them causes bad designs.

You can raise throughput almost indefinitely by adding parallelism: more servers, more partitions, more consumers. Throughput is largely a resource question. Latency is not. A single request has a critical path, and no amount of horizontal scale makes that path shorter. If placing an order requires a synchronous call to a payment gateway that takes 300 milliseconds, then placing an order takes at least 300 milliseconds no matter how many servers you own. To make it faster you have to change the path itself: cache something, parallelise independent steps, or move work off the critical path into an asynchronous flow. That last option is the entire subject of the API series, and it is the deepest lever in this pillar.

Every request also has a latency budget, and the budget is a design input, not an afterthought. "The order confirmation screen must appear within 500 milliseconds" tells you how much of that 500 milliseconds each step on the critical path is allowed to consume, and which steps have to come off the path entirely. Without a budget, you cannot tell whether a design is good enough. With one, the design pressures itself.

Averages Hide the Failures That Matter

The last habit is to stop looking at averages. Average latency is a comforting number that describes an experience almost none of your users have.

What users feel is the tail. If your average response time is 80 milliseconds but your 99th percentile is 4 seconds, then one request in a hundred is a bad experience, and at 20,000 requests per second that is 200 bad experiences every second. Worse, the tail compounds with fan-out. If a single user action makes ten downstream calls and each has a 1-in-100 chance of being slow, the probability that at least one of the ten is slow is closer to 1-in-10. Tail latency at the component level becomes typical latency at the user level. This is why high-fan-out systems live or die on their p99 and p999, not their average.

So the throughput target should always arrive with a latency distribution attached: not "under 100 milliseconds" but "p50 under 50 milliseconds, p99 under 200 milliseconds, p999 under 1 second." Those three numbers describe a system. One average describes nothing you can design against.

Know Your Numbers Before You Design

Everything above reduces to a single discipline: know your numbers before you design against them, and distrust any number that arrives alone.

Before drawing a box, be able to fill in a short table for the system you are scaling:

You will not always have exact figures early, and estimates are fine. What is not fine is a design that never named the figures at all, because such a design is optimising for a number that means nothing. Almost every expensive scaling mistake I have seen traces back to a team that scaled the wrong half of the system, because they were looking at the headline instead of the decomposition.

Conclusion

A throughput target is a headline, not a specification. Reading it honestly means decomposing it into the properties that actually drive design: peak versus sustained, the read/write mix, the fan-out that turns one action into many, the difference between throughput and latency, and the tail behaviour that averages conceal. Do that decomposition first and every later decision in this pillar has something real to push against.

Part 2 takes the food-delivery system and walks it up the orders of magnitude, from hundreds of requests per second to millions, naming what breaks first at each rung and the cheapest thing that fixes it. The numbers we learned to read here are what tell us which rung we are standing on.