Published on

Scaling the API: Infra and Operating at Scale

Authors

This is Part 5 of Scaling: The API / Backend Track, the second series in a pillar on scaling systems.

The previous parts scaled the flow of work. Part 5 is about the ground it all runs on and the discipline of operating it. We started this series by naming the database write ceiling as the real limit of synchronous scaling; here we finally go through it, with sharding. Then autoscaling, backpressure, and the two cross-cutting concerns, security and observability, that determine whether an asynchronous system is something you can actually run at 3am or merely something you can draw on a whiteboard. This part closes the backend track and hands off to the frontend.

Table of Contents

Sharding: Past the Single-Primary Ceiling

Part 1 named the ceiling: reads scale by copying, but every write lands on one authoritative primary, and one machine has a finite durable write rate. Caching and replicas do nothing for writes. When sustained writes exceed what one primary can commit, there is one structural way through, and it is sharding: stop having one database, and split the data across many, each owning a slice and each accepting writes for its slice only.

Split the food app's orders across, say, sixteen shards by customer, and each shard sees roughly a sixteenth of the write load. The single-primary ceiling is gone, because there is no single primary; write capacity now scales with shard count, the same way the stateless service tier scaled with instance count. This is how you scale state horizontally, and it is the most powerful and most expensive tool in the data layer.

Expensive, because sharding complicates everything that was simple with one database:

  • Cross-shard queries. "All orders in the last hour" now has to fan out to every shard and merge the results, rather than being one query. Anything that spans shards is harder, slower, or both.
  • Cross-shard transactions. A transaction touching two shards is a distributed transaction, which is exactly the kind of cross-system atomicity Part 4 showed you cannot get cheaply. In practice you design hard to keep related data on the same shard so you rarely need one.
  • Rebalancing. When a shard gets too hot or you add capacity, moving data between shards while the system is live is a delicate operation in its own right.
  • Operational surface. Sixteen databases are more to back up, monitor, patch, and reason about than one.

Sharding is therefore a lever you pull when the write ceiling forces you to, not before. Everything in Part 1, statelessness, caching, replicas, exists partly to delay this day, because the day you shard is the day your data layer gets structurally more complex forever.

Choosing a Shard Key

If sharding has one make-or-break decision, it is the shard key, because the key decides how evenly load spreads and how often you have to cross shards, and it is extremely painful to change later. Two properties matter most.

The key must distribute load evenly. A key that clumps traffic creates a hot shard that hits the single-machine ceiling while the others idle, which defeats the entire point. Sharding by country sounds natural and is usually a trap, because one large market can dwarf the rest and land on one overloaded shard. A high-cardinality key like customer ID spreads far more evenly, since customers are numerous and roughly balanced.

The key should also keep related data together, so that your common queries and transactions stay within a single shard. If you shard orders by customer, then "this customer's order history" is one shard's local query and any transaction over a customer's own data stays local, which is exactly what you want. The art is finding a key that is both well-distributed and aligned with your access patterns; customer ID often satisfies both for consumer systems, which is why it is such a common choice. The lesson echoing the whole pillar: the shard key is a long-lived, load-shaping commitment, so derive it from your real access patterns and volumes rather than picking the first field that seems unique.

Autoscaling the Right Thing

Autoscaling adds and removes capacity as load changes, and Part 1 of the first series is why it matters: bursty, peak-heavy traffic like the dinner rush is wasteful to serve with a fleet fixed at peak size and fatal to serve with one fixed at average. Autoscaling lets capacity track demand.

The trap is autoscaling the wrong tier. The stateless service tier autoscales beautifully, add instances when CPU or request rate climbs, remove them when it falls, because, as we keep returning to, stateless things scale freely. But blindly scaling the service tier can actively hurt, because more app instances mean more database connections (the Part 1 pooling ceiling) and more write pressure on a primary that cannot scale the same way. Autoscaling the front while the real bottleneck is the data layer just drives more traffic into the wall.

So autoscale with the bottleneck in mind. Scale the stateless tier on its own signals, but respect the downstream limits: cap connections, and recognise that past a point the answer is not more app servers but more shards or more cache. On the consumer side, Kafka consumers autoscale within the hard limit from Part 3, you can add workers only up to the partition count, so scaling consumers and provisioning partitions are the same capacity conversation. Effective autoscaling is not "scale everything on CPU"; it is scaling each tier on the signal that reflects its actual constraint.

Backpressure: Degrade, Do Not Collapse

No matter how you scale, demand can exceed capacity, a viral moment, a regional surge, a dependency slowdown. What happens next is decided by whether you built backpressure, and it is the difference between a degraded evening and an outage.

Without backpressure, an overloaded system does not slow down gracefully; it collapses. Queues grow without bound, memory fills, latencies climb until timeouts cascade, and the system falls over entirely, often taking longer to recover than the spike itself lasted. Backpressure is the deliberate design of what to do when you cannot keep up, choosing to degrade on purpose rather than die by accident:

  • Rate limiting caps accepted work at a sustainable level and rejects or delays the excess, protecting the core. Better to cleanly refuse some requests than to accept everything and serve none.
  • Load shedding drops the least important work first under stress, keeping order placement and payment alive while pausing recommendations or non-urgent notifications.
  • Bounded queues and buffers. The broker itself is a shock absorber, from Part 2's load-leveling, but every internal queue needs a ceiling and a defined behavior when full, so that a backlog somewhere does not consume all memory everywhere.

The mindset is to decide your degradation strategy before overload, not during it. A system that sheds load deliberately keeps its critical paths alive through a spike; a system that never considered overload discovers its degradation strategy at the worst possible moment, and it is usually "fall over."

Security at Scale

Security is a cross-cutting concern the whole way up, and scale changes both the threat and the mechanisms. Two aspects bear directly on scaling.

Authentication and authorization have to scale with the traffic, which means they cannot depend on a central lookup on every request. Checking a session against a central store on all twenty thousand requests per second makes that store the bottleneck and single point of failure. The scalable pattern is self-contained, verifiable credentials, signed tokens that carry the user's identity and claims, so any stateless service validates the token locally without a round trip. This is a direct consequence of the statelessness that made the service tier scale in the first place: stateless auth is what lets stateless services stay stateless.

Rate limiting and abuse protection are load-management as much as security. At scale you face scrapers, credential-stuffing, and outright denial-of-service attempts, and the same rate-limiting machinery that provides backpressure also blunts abuse, keeping malicious or runaway traffic from consuming the capacity real users need. The two motivations converge on the same control at the edge. Security at scale is less about new secrets and more about mechanisms that hold up under volume without becoming bottlenecks themselves.

Observability: Operating the Invisible

The deepest operational cost of going asynchronous, flagged back in Part 2, is that you can no longer follow a request as a straight line, and that makes observability not a nice-to-have but the precondition for operating the system at all.

In a synchronous world a failure is one stack trace down one call chain. In the async world we have built, a single order flows from a producer, through Kafka, to several consumers running at different times, possibly through a saga with compensations, across sharded databases. When something goes wrong, "where did order 123 get stuck" is a genuine investigation, and without the right telemetry it is an impossible one. Three pillars make it tractable:

  • Distributed tracing. A trace ID attached to the order and propagated through every service, event, and consumer lets you reconstruct the whole journey across async hops. In an event-driven system this is not optional; it is the only way to answer "what happened to this order," and it must be designed in from the start because retrofitting it is painful.
  • Metrics. Rates, error rates, and latencies per service, plus the async-specific ones that reveal health, consumer lag (how far behind real time each consumer is), queue depths, dead-letter volume, and rebalance frequency from Part 3. Consumer lag in particular is the vital sign of an async backend: rising lag means consumers are losing the race with producers, the earliest warning that something is wrong.
  • Logs. Structured and correlated by the same trace ID, so that the detailed record of any one order can be pulled together from across all the services that touched it.

The blunt truth is that an asynchronous system without strong observability is not operable. You will not be able to debug it, and you will not be able to trust it. So the observability is part of the cost of going async, to be budgeted alongside the broker itself, not bolted on after the first incident you could not explain.

Conclusion, and the Handoff

Operating a system at scale is the infrastructure beneath the flow. Sharding takes the data layer through the single-primary write ceiling, at the price of permanent complexity, so you do it only when the writes force you to and you choose the shard key with great care. Autoscaling tracks bursty demand but must respect the real bottleneck rather than blindly scaling the front. Backpressure ensures that when demand exceeds capacity the system degrades on purpose instead of collapsing by accident. And security and observability are the cross-cutting concerns that make an asynchronous system safe and operable, stateless auth that scales with traffic, and tracing and consumer-lag metrics that make the invisible flow visible.

That closes the backend track. We have taken the sync-versus-async spine from "how far does request-response go" all the way to "how do you operate a sharded, event-driven system in production." But every one of these backend decisions, especially the move to eventual consistency, eventually surfaces on a screen in front of a real person. The UI / Frontend Track picks up exactly there: how to serve clients at scale, and how the asynchronous, eventually-consistent backend we just built has to be represented honestly in the user experience. The work is only half done until the client tells the truth about it.