Published on

Scaling the API: How Far Request-Response Actually Goes

Authors

This is Part 1 of Scaling: The API / Backend Track, the second series in a pillar on scaling systems. The first series, Orders of Magnitude, built the mental model: read the numbers, walk the ladder, survey the levers, choose under constraints. This series takes the sync-versus-async spine all the way down.

The async parts of this series get the attention, because they are interesting. But reaching for asynchronous processing before you have exhausted synchronous scaling is one of the most common and most expensive mistakes in the field. So we start where you should start: with a clear-eyed account of how far plain request-response actually goes, which is much further than its reputation suggests. Our worked example remains the food-delivery app and its "place order, take payment, confirm" synchronous path.

Table of Contents

In Defense of the Boring Path

Synchronous request-response is the default for good reason. The client asks, the server answers, and the answer either succeeded or it did not. The whole interaction is a straight line you can read top to bottom, reason about, log, trace, and debug. When something goes wrong, the error comes straight back to the caller who can react to it immediately. Consistency is easy, because the caller does not get an answer until the work is actually done.

Every one of those properties is something you give up the moment you go asynchronous, and we will spend the next parts accounting for the loss. So the bar for leaving the synchronous path should be high, and the first question when someone proposes a broker should always be: have we actually run out of room on the simple path, or does it just feel unsophisticated? Usually there is far more room than the proposer thinks, and it is found in four boring, well-understood techniques.

Stateless Services Scale for Free

The most important property a synchronous service can have is statelessness, because it is the property that makes all the cheap scaling possible.

A stateless service keeps nothing in local memory that a later request depends on. Every request carries or fetches everything it needs: the user's identity comes from a token, the session lives in a shared store, the data lives in the database. Because no server holds anything special, any server can handle any request, and that single fact is what makes horizontal scaling trivial. Need more capacity? Run more identical instances behind a load balancer. Twenty instances handle roughly twenty times the load of one, and adding the twenty-first is a non-event.

This is why, in Orders of Magnitude, the compute tier scaled almost for free all the way up the ladder. Stateless application servers are the easy part of scaling. If you preserve statelessness, the synchronous service tier will rarely be your bottleneck, because you can always add more of it. The bottleneck lives one layer down, in the shared state that all those identical servers are talking to.

Connection Pooling: The Quiet Ceiling

There is one ceiling on the service tier that surprises teams because it appears suddenly and at the worst possible time, at peak, and it comes from connection pooling.

Each application server holds a pool of open connections to the database, reusing them across requests because opening a fresh connection per request is far too slow. This is correct and necessary. The trap is arithmetic. A database can only tolerate so many concurrent connections, each one costs it memory and scheduling overhead, and that limit is often lower than people expect, in the low thousands for many configurations. Now run fifty stateless app servers, each with a pool of a hundred connections, and you have asked for five thousand connections from a database that will accept two thousand. It works fine in testing with three servers. It falls over the first evening you scale out to fifty, and the error, connections refused, points at the database while the real cause is the fleet size times the pool size.

The fix is to treat total connections as a budget: pool size times instance count must stay under what the database allows, which often means smaller per-instance pools than feel natural, or a connection proxy that multiplexes many app-side connections onto few database ones. The point for this series is subtler: even the "free" stateless tier has a hidden coupling to the shared datastore, and that coupling is, again, about state. The database keeps showing up as the constraint.

Caching in Front of the Database

Once the service tier scales out, database load becomes the story, and the highest-leverage response is caching, because it attacks the read/write asymmetry head on.

In the food app, the same restaurant lists and menus are read thousands of times per second and change rarely. Without a cache, every one of those reads is real work against the database, work that produces the identical answer over and over. Put a cache in front, an in-memory store the app checks before the database, and the hot reads are served from memory in a fraction of the time and at a fraction of the cost. The database load for those paths does not shrink a little, it collapses, because you have stopped asking the same question thousands of times a second.

The cost, as always, is correctness under change. A cache is a second copy of the truth, so the moment the truth changes the copy is stale, and you have to decide how much staleness you can tolerate and how you invalidate. For slow-changing catalog data the answer is easy: a short time-to-live is fine, because a menu that is thirty seconds out of date harms no one. For a live order status it is harder, and that difficulty is a hint that some data does not belong in a simple cache at all. But for the large, read-heavy, slow-changing majority of traffic, caching is the single most effective lever in synchronous scaling, and it should usually be pulled before anything more exotic is even discussed.

Read Replicas

Caching handles the reads that repeat. Read replicas handle the reads that do not repeat but can tolerate being slightly behind.

A replica is a live copy of the database that receives a stream of changes from the primary and serves reads. Point your read-heavy, non-cacheable queries, a customer's order history, past receipts, account details, at replicas, and you add read capacity by adding copies, keeping that load off the primary entirely. Like caching, replicas exploit the fact that reads do not have to agree with each other in real time; unlike caching, they serve arbitrary queries rather than pre-computed hot answers.

The cost is replication lag. A replica is always a little behind the primary, usually milliseconds, occasionally more under load, so a read from a replica may not reflect a write that just happened. This produces the classic "I placed an order but it is not in my history yet" glitch when the write hit the primary and the read hit a replica that had not caught up. The fix is to route reads that must see the latest write, the "read your own writes" cases, to the primary, and everything else to replicas. Getting that routing right is real work, but it lets a single logical database serve read volumes far beyond what one machine could, without leaving the synchronous model at all.

The Database Is the Real Ceiling

Stack up the techniques and a pattern is unmistakable. Stateless services scale for free. Connection pools are bounded by the database. Caching offloads the database. Replicas offload the database. Every technique in synchronous scaling is, in the end, about protecting one thing: the primary database, and specifically its ability to accept writes.

This is the ceiling. You can cache and replicate reads without limit, but every write in the system ultimately lands on one authoritative primary, and that primary is one machine with one disk and a finite rate at which it can durably commit. When the sustained write rate approaches that limit, none of the read-side techniques help, because you cannot cache a write and you cannot replicate your way to more write capacity. This is the same lesson Orders of Magnitude reached from the other direction: state is what hurts, and the write ceiling is where state hurts most.

There are exactly two ways past this ceiling, and they are the subjects of the rest of the series. You can split the writes across many databases so no single primary sees all of them, which is sharding (Part 5). Or you can stop demanding that all the work happen synchronously before you answer, moving the deferrable writes off the critical path into an asynchronous flow (Parts 2 through 4). Most high-scale systems do both. But you only need either one once you have genuinely hit the write ceiling, and knowing where that ceiling is means you can tell the difference between needing a broker and merely wanting one.

When Sync Is Genuinely Enough

So when is plain request-response actually enough? More often, and for longer, than the enthusiasm for distributed systems suggests.

If your writes fit comfortably on one well-provisioned primary, if your reads are served by caches and replicas, and if your latency budget is met on the synchronous critical path, then you are done, and adding a broker would only add cost, latency, and operational burden for no benefit. Plenty of successful systems serving very real traffic run on exactly this: stateless services, aggressive caching, a few read replicas, and one primary database that has not yet been asked for more writes than it can give. That is not an embarrassing architecture to be scaled away from at the first opportunity; it is frequently the correct architecture, and the discipline is to keep it until the numbers say otherwise.

The signal to move is specific and measurable, not a feeling. It is a write rate that a single primary cannot sustain, or a synchronous chain whose coupling is now causing cascading failures at peak, exactly the crack we will pull apart in Part 2. Until you see that specific signal in your own metrics, the boring path is the right path.

Conclusion

Synchronous request-response is the default because it is simple, consistent, and debuggable, and it scales much further than its reputation. Stateless services scale for free, so the service tier is rarely the bottleneck. Connection pooling, caching, and read replicas each buy substantial headroom, and every one of them is really a way of protecting the primary database, because the database write ceiling is the real limit of the synchronous model.

You leave the synchronous path when, and only when, you hit that ceiling: when writes exceed what one primary can hold, or when synchronous coupling starts causing failures at scale. Part 2 is about that exact moment, the point where sync breaks and a broker like Kafka finally earns its place, and about the honest bill that comes with it.