Published on

Scaling the API: Where Sync Breaks, and the Broker Enters

Authors

This is Part 2 of Scaling: The API / Backend Track, the second series in a pillar on scaling systems.

Part 1 made the case that synchronous request-response scales further than people think, and named the exact ceiling it eventually hits. Part 2 is about that moment. It is the heart of the whole pillar: where synchronous coupling breaks under scale, why a broker like Kafka is the answer, and, just as important, the full price of that answer. Reach for a broker for the right reasons and it transforms a system. Reach for it as a fashion and it buries you in complexity you did not need.

Table of Contents

How the Synchronous Chain Cracks

Recall the fan-out from the first series: one "Place Order" tap becomes validate cart, check restaurant, reserve inventory, authorise payment, create order, notify restaurant, enqueue courier assignment, write audit events. In the synchronous design, the order request calls all of these in a chain and waits for each before answering the customer.

At low volume this is fine. At high volume it cracks along two seams.

The first is availability by multiplication. A synchronous chain is only as available as the product of its links. If each of seven downstream services is up 99.9 percent of the time, the chain is up only about 99.3 percent of the time, because any one failure fails the whole request. You have taken seven pretty-reliable services and composed them into one noticeably-unreliable order path, purely by demanding they all succeed together, right now.

The second, and more violent, is latency coupling under load. Suppose the notification service gets slow, taking two seconds instead of fifty milliseconds. In a synchronous design, every order request now holds a thread and a connection open for two seconds waiting on notification, work that has nothing to do with whether the order was placed. Those held resources pile up, the connection pool from Part 1 exhausts, and new order requests are refused. A slowdown in a non-critical service, sending a notification, has taken down order placement itself. This is cascading failure, and it is the characteristic way synchronous systems die at scale. The problem is not any one service; it is that everything is bolted rigidly to everything else.

The key realisation is that most of those downstream steps do not need to happen before you answer the customer. The customer needs to know the order was accepted and paid. Whether the restaurant has been pinged yet, whether a courier has been assigned yet, whether the receipt email has gone out, none of that has to be true at the moment of confirmation. It only has to become true, reliably, soon. That gap between "must be true now" and "must become true reliably soon" is the entire opening for asynchronous processing.

The Broker in One Idea

A message broker splits a synchronous call into two halves that no longer wait for each other. Instead of the order service calling the notification service and waiting, the order service writes an event, "order 123 was placed", to the broker and moves on immediately. The notification service reads that event from the broker, on its own schedule, and does its work. The broker sits in the middle and holds the event durably until the consumer is ready.

Kafka is the canonical example, and it is more precisely a distributed, durable, append-only log. Producers append events to the end; consumers read forward at their own pace, each tracking its own position. That log-shaped design is what gives Kafka its particular strengths, and Part 3 gets into the mechanics. For now the idea is enough: the producer and consumer are decoupled in time by a durable buffer between them. Everything good and everything costly about async flows from that one structural change.

Good Reason 1: Decoupling

The first and deepest reason is decoupling, and it directly cures the cascading failure above.

Once the order service only writes an event and returns, it no longer depends on the notification service being up or fast. If notification is down, events pile up harmlessly in the broker and get processed when it recovers; order placement stays fast and available the entire time. The availability-by-multiplication problem dissolves, because the services no longer have to all succeed in the same instant. The latency-coupling problem dissolves too, because a slow consumer just falls behind in the log rather than holding the producer's resources hostage.

Decoupling also loosens the human coupling. New consumers can subscribe to "order placed" without the order service knowing or changing, so the analytics team, the loyalty team, and the fraud team can each read the same events independently. The producer publishes facts; consumers decide what to do with them. That is a fundamentally more scalable way to grow a system than wiring every new feature directly into the order service's synchronous path.

Good Reason 2: Load-Leveling

The second reason is absorbing spikes. A broker is a buffer, and a buffer turns a spike into a queue instead of a collapse.

The food app's load is bursty by nature, quiet all afternoon, then a wall at dinner. A synchronous downstream service sized for the average gets overwhelmed at peak and starts failing; sized for the peak, it sits idle and expensive all day. A broker breaks the dilemma. Producers write events as fast as they arrive during the dinner spike, and the broker holds them. Consumers process at their own steady, sustainable rate, draining the backlog. The peak becomes a temporary deepening of the queue rather than an overload, and the consumer fleet can be sized for sustainable throughput rather than instantaneous peak. You trade a little latency during the spike, events wait their turn, for the ability to survive the spike at all without over-provisioning. This is often reason enough on its own.

Good Reason 3: Fan-Out

The third reason is one-to-many delivery. One event, many independent consumers, each reading the same log without interfering with the others.

"Order placed" is interesting to a lot of parts of the business: notifications, the restaurant dashboard, courier dispatch, analytics, fraud detection, loyalty points. In the synchronous world, the order service would have to call every one of them, growing more coupled and more fragile with each new consumer, and slower, because it waits on all of them. With a broker, the order service writes the event once and every interested consumer reads it independently and at its own pace. Adding the seventh consumer costs the producer nothing and risks nothing. This is how event-driven systems accommodate growth gracefully where synchronous ones accumulate coupling until they seize.

Good Reason 4: Durability and Replay

The fourth reason is subtler and is specific to log-shaped brokers like Kafka: the event history persists, and that unlocks capabilities a synchronous call simply does not have.

Because the log durably retains events, a consumer that was down does not lose anything; it resumes from where it left off and catches up. A brand-new consumer can start from the beginning and process all of history, which is how you backfill a new feature against past events. If a consumer had a bug that mis-processed a week of orders, you can fix it and replay that week. The event log becomes a durable record of what happened, not just a transient pipe, and "replay the events" becomes a routine and powerful operational tool. Synchronous calls vanish the instant they complete; a durable log remembers, and that memory is worth a great deal when things go wrong, as they will.

The Honest Bill

Everything above is real, and none of it is free. Going asynchronous is one of the biggest bills in the pillar, and paying it knowingly is what separates good use of a broker from cargo-culting it.

You gainYou pay
Decoupling, availabilityEventual consistency: the work is not done when you reply
Spike absorptionAdded end-to-end latency for the deferred work
Fan-out to many consumersA new distributed system to operate: the broker itself
Durability and replayMuch harder debugging: the flow is no longer a straight line
Independent scalingNew correctness problems: ordering, duplicates, partial failure

The heaviest item is the first. In a synchronous design, when you told the customer the order was placed, everything was done. In an async design, "order placed" means the order is recorded and the rest will happen soon, but has not happened yet. The restaurant has not been notified in this instant; the courier is not assigned yet. Your system is now eventually consistent, and that reality does not stay in the backend. It propagates all the way to the user interface, where the client has to represent work that is genuinely in progress. That propagation is the entire subject of the UI series.

The other items are just as real. The broker is a distributed system your team now runs and gets paged for. Debugging changes character completely: instead of reading one stack trace down a synchronous call, you are correlating events across producers, the log, and multiple consumers running at different times, which is why distributed tracing stops being optional (Part 5). And async introduces failure modes that synchronous code never had, duplicate deliveries, out-of-order events, half-finished multi-step flows, each of which has to be handled deliberately. Part 4 is devoted entirely to that bill, because it is the part teams most often underestimate.

When NOT to Reach for a Broker

Because the bill is so high, the most valuable judgment is knowing when not to pay it. Reach for a broker when you have the problems it solves. Do not reach for it otherwise. Concretely, you probably should not introduce one when:

  • The work genuinely must be synchronous. If the caller cannot proceed without the result, taking a payment before showing a confirmation, checking inventory before accepting an order, then it belongs on the synchronous path. Do not make a fundamentally synchronous interaction asynchronous just to use a broker; you will only bolt a confirmation-polling mechanism back on top and reinvent request-response, worse.
  • You have not hit the synchronous ceiling from Part 1. If your writes fit on one primary and your chain is not cascading at peak, a broker adds cost and complexity for no benefit. Solve the problem you have.
  • You need a simple work queue, not an event platform. If all you want is to run a handful of background jobs, a plain job queue may give you what you need at a fraction of Kafka's operational weight. Kafka earns its keep with high throughput, many consumers, and replay; for a modest background task it can be overkill.
  • Your team cannot operate it. As the first series argued, the best architecture your team cannot run is worse than a simpler one it can. A broker is a serious operational commitment, and taking it on before you can support it trades a scaling problem you do not yet have for a reliability problem you will have immediately.

The discipline is the same one that runs through the whole pillar: match the tool to the actual problem and the actual scale, not to the sophistication you wish the system had.

Conclusion

The synchronous chain cracks at scale along two seams: availability that degrades as failures multiply, and latency coupling that turns one slow service into a system-wide collapse. A broker cures both by decoupling producers from consumers through a durable buffer, and it throws in load-leveling, fan-out, and replay. Those are the right reasons to adopt it.

The bill is eventual consistency, added latency, a new system to operate, much harder debugging, and a fresh class of correctness problems, and it is high enough that "when not to" is as important as "when." Part 3 opens up the broker itself, the partitions, keys, consumer groups, and delivery semantics you have to understand to reason about scale, before Part 4 confronts the correctness bill head on.