Published on

Scaling the API: Correctness in an Async World

Authors

This is Part 4 of Scaling: The API / Backend Track, the second series in a pillar on scaling systems.

Part 3 ended on a stance: run at-least-once, engineer for idempotency. Part 4 makes good on it. This is the part of asynchronous systems that teams most consistently underestimate, the correctness bill from Part 2 come due. In a synchronous world, the caller waits and gets a clear success or failure, and correctness is mostly free. In an async world, messages get delivered twice, consumers crash mid-work, a write to the database and a publish to the broker can disagree, and a multi-step flow can strand halfway. None of these are exotic; at scale they happen constantly. This part is the toolkit for staying correct anyway.

Table of Contents

Why Async Breaks Correctness

Three properties of asynchronous systems, each introduced earlier, combine into a correctness problem.

First, at-least-once delivery means duplicates. From Part 3, the pragmatic default guarantees no message is lost but allows the same message to be delivered more than once. So every consumer must assume it will occasionally see the same event twice.

Second, failures are partial and invisible to the caller. The producer wrote the event and moved on; it is not waiting to hear whether the consumer succeeded. If the consumer crashes halfway through, nobody is standing there to react. Recovery has to be built into the system rather than handled by the caller.

Third, there is no shared transaction across the boundary. The producer's database and the broker are two separate systems. You cannot wrap "write the order to the database" and "publish the order-placed event" in one atomic transaction across both, which opens the door to them disagreeing.

Put together, these mean correctness in async is not automatic and not optional; it is a set of patterns you apply deliberately. The rest of the part is those patterns, in the order you should reach for them.

Idempotency: The Foundation

Idempotency is the foundation, and if you take one thing from this part, take this: an operation is idempotent if performing it twice has the same effect as performing it once. Once your processing is idempotent, duplicate delivery stops being a correctness problem and becomes a harmless inefficiency, and that single property is what makes at-least-once delivery safe to build on.

The pattern is to give each unit of work a stable identity and to record what you have already done. Concretely, in the food app, the "charge payment for order 123" event carries a unique idempotency key, often the order ID or a dedicated payment ID. Before charging, the payment consumer checks whether it has already processed that key. If yes, it does nothing and reports success; if no, it performs the charge and records the key atomically with the charge. Now if the event is delivered twice, the second attempt sees the key already processed and skips, and the customer is charged exactly once even though the event arrived twice.

The subtlety is that the check-and-record must be atomic with the effect, or you reopen the very gap you are closing. If you charge the card and then, in a separate step, record the key, a crash in between leaves the card charged and the key unrecorded, and the retry charges again. The usual solutions are a unique constraint in the database that makes a duplicate insert fail, or passing the idempotency key through to a downstream, payment gateways typically accept an idempotency key precisely for this reason, so that the external system enforces once-only for you. Idempotency is not glamorous, but it is the property everything else in this part leans on.

Retries and Their Discipline

Because failures are partial and unsupervised, retrying is how async systems recover, and idempotency is what makes retrying safe. But retries need discipline, because naive retries make outages worse.

Three rules keep retries healthy:

  • Back off exponentially. If a downstream is failing, retrying immediately and forever just piles load onto something already struggling, turning a blip into a sustained outage. Wait longer between each attempt, and add jitter, small randomness, so that a thousand consumers do not all retry in lockstep and hammer the recovering service in synchronized waves.
  • Cap the attempts. Some failures are transient and clear on retry; some are permanent, a malformed event, a business rule that will never pass, and retrying those forever is pure waste that clogs the pipeline. After a bounded number of attempts, stop and set the message aside, which is what dead-letter queues are for.
  • Distinguish retryable from terminal failures. A network timeout is worth retrying; a validation error saying the order references a restaurant that does not exist is not, and will fail identically every time. Retrying terminal failures wastes resources and delays the messages behind them. Where you can, classify the failure and route accordingly.

Retries plus idempotency are the everyday mechanism by which an async system heals from the constant small failures of a distributed world. Retries plus non-idempotent processing are how you double-charge customers. The two belong together.

Dead-Letter Queues

A message that has failed its maximum retries cannot block the pipeline forever, and it cannot be silently dropped either, because it might be a real order or a real payment. The answer is the dead-letter queue: a separate place where messages that could not be processed are set aside for later inspection.

The pattern is simple and the discipline around it is what matters. When a message exhausts its retries, move it to the dead-letter queue and let the main pipeline continue with the next message, so one poison message does not stall everything behind it. Then, crucially, treat the dead-letter queue as an operational surface, not a graveyard. It needs monitoring and alerting, because messages landing there mean something is wrong, a bug, a bad deploy, a downstream that changed its contract. Someone has to look, diagnose, fix, and often replay the messages back through once the fix is in. A dead-letter queue nobody watches is just a place where orders go to disappear quietly, which is worse than a loud failure. Watched, it is the safety net that lets the rest of the system keep flowing while genuine problems get human attention.

The Dual-Write Problem and the Outbox Pattern

Now the subtle one, the third property from the start of the part: there is no transaction spanning your database and the broker, and that creates the dual-write problem.

Consider what "place an order" has to do: write the order to the database, and publish an "order placed" event to Kafka. These are two systems, and you cannot make both happen atomically. So there is a gap:

  • If you write to the database first and then publish, a crash in between leaves an order that exists but was never announced. The restaurant is never notified, no courier is assigned, the order silently strands.
  • If you publish first and then write, a crash in between announces an order that does not exist. Consumers act on a phantom, and the database has no record of it.

Either ordering can leave the two systems disagreeing, and at scale, over millions of orders, "rare crash in between" is a guaranteed daily occurrence. Retries do not save you here, because the problem is that one of the two writes never happened at all.

The standard solution is the outbox pattern, and it is worth knowing because the problem is so common. Instead of writing to the database and separately publishing, you write both the order and the event into the same database, in one transaction, the event going into an "outbox" table. Because they are in one database, that write is atomic: either both the order and the outbox row commit, or neither does. A separate process then reads new outbox rows and publishes them to Kafka, marking each as published once the broker acknowledges. If that publisher crashes, it resumes from the unpublished rows on restart, so every committed order's event is eventually published, exactly the at-least-once guarantee we already know how to handle with idempotency downstream. The outbox turns an impossible cross-system atomic write into an ordinary single-database transaction plus a reliable relay, and it is the canonical answer to keeping your database and your broker in agreement.

Sagas: Correctness Across Steps

The hardest correctness problem is a business operation that spans several services and cannot be wrapped in one transaction, and placing a paid order is exactly that: reserve inventory, charge the payment, create the order, assign a courier. In a single database you would make these one atomic transaction that either wholly succeeds or wholly rolls back. Across services and a broker, no such transaction exists. So what happens when the payment succeeds but courier assignment then fails? You have taken the customer's money for an order that cannot be delivered.

The saga pattern is the answer: model the multi-step operation as a sequence of local steps, each with a compensating action that undoes it. Instead of one big transaction that rolls back, you get a chain of small committed steps, and if a later step fails, you run the compensations for the steps already done, walking the operation backward. If courier assignment fails after payment succeeded, the saga triggers a refund, the compensation for the charge, and cancels the order. The system reaches a consistent end state, order cancelled and money returned, not through rollback but through deliberate reversal.

Sagas come in two flavors worth naming. In a choreographed saga, each service listens for the previous step's event and emits its own, with no central coordinator; it is loosely coupled but the overall flow is implicit and can be hard to follow. In an orchestrated saga, a coordinator explicitly drives the steps and invokes compensations; it is easier to reason about and monitor at the cost of a central component. For a critical, auditable flow like payment, orchestration's visibility is usually worth it. Either way, the essential idea is the mental shift: at scale, across services, you do not get atomic all-or-nothing transactions, so you design for consistency through compensation instead. That shift is one of the defining differences between building at small scale and building at large scale.

The Domain Lens: How Much Rigor

Every pattern here costs effort, and how much of it you apply is set by the domain, the same lens the first series drew. The mechanics are universal; the required rigor is not.

A bank applies all of it, without compromise, on the money path. Exactly-once effects via strict idempotency, carefully ordered sagas with guaranteed compensation, meticulously monitored dead-letter queues, and an audit trail of every step, because a lost or duplicated transaction is a catastrophe and a regulatory event. The cost in complexity and latency is simply accepted; correctness on the ledger is non-negotiable.

A food app is more relaxed, deliberately. It still wants idempotency on the payment, nobody may be double-charged, so that path gets bank-like care. But a duplicated push notification is a shrug, and a status update that arrives a little out of order self-corrects when the next one lands, so those paths can run with lighter guarantees and simpler handling. The engineering judgment is to spend your correctness budget where being wrong is expensive and to relax where it is cheap. That judgment, not the patterns themselves, is what experience buys: knowing that the payment saga deserves orchestration and audit while the notification consumer just needs a best-effort retry.

Conclusion

Going asynchronous trades the easy correctness of synchronous request-response for a set of problems you must handle on purpose: duplicates from at-least-once delivery, partial failures with no caller waiting to react, and the impossibility of a transaction spanning your database and the broker. The toolkit answers each. Idempotency is the foundation that makes duplicates harmless. Retries with backoff heal transient failures, and dead-letter queues catch what cannot be healed. The outbox pattern keeps the database and the broker in agreement, and sagas keep multi-step operations consistent through compensation rather than rollback.

How much of this rigor you apply is a domain decision, full bank-grade care on the payment path, lighter touch on notifications. Part 5 closes the backend series with the infrastructure that carries all of this at scale: sharding the data layer, autoscaling, backpressure end to end, and the security and observability that let you actually operate an asynchronous system in production.