Scaling the API: Kafka Mechanics That Matter for Scale

This is Part 3 of Scaling: The API / Backend Track, the second series in a pillar on scaling systems.

• Part 1: How Far Request-Response Actually Goes: the honest defense of synchronous request-response.
• Part 2: Where Sync Breaks, and the Broker Enters: the real reasons to reach for Kafka, and the honest bill.
• Part 3: Kafka Mechanics That Matter for Scale: partitions, keys, consumer groups, and delivery semantics.
• Part 4: Correctness in an Async World: idempotency, retries, dead-letter queues, the outbox, and sagas.
• Part 5: Infra and Operating at Scale: sharding, autoscaling, backpressure, security, and observability.

Part 2 argued for the broker in the abstract. Part 3 opens it up just enough to reason about scale. This is not a Kafka tutorial, there are excellent ones, and the operational details shift between versions. It is the mental model of the few mechanics that actually govern throughput, ordering, and correctness, so that when you size a system or debug a backlog you know which knob you are turning and what it costs. Everything here is in service of the correctness work in Part 4 and the operations in Part 5.

Table of Contents

• The Log, Precisely
• Partitions: The Unit of Parallelism
• Keys and Ordering
• Consumer Groups: Dividing the Work
• Why Partition Count Bounds Throughput
• Rebalancing and Its Cost
• Delivery Semantics: At-Least-Once vs Exactly-Once
• Conclusion

The Log, Precisely

A Kafka topic is an append-only log. Producers append records to the end; consumers read forward and remember their position, called the offset. Records are not deleted when read, they persist for a configured retention period, which is why multiple independent consumers can read the same topic and why replay is possible, as Part 2 described.

The single most important thing to internalise is that a topic is not one log but several. Each topic is split into partitions, and each partition is an independent, ordered log. This one design choice is the source of nearly every scaling property and nearly every constraint that follows. Almost every question about Kafka performance or ordering resolves to a question about partitions.

Partitions: The Unit of Parallelism

Partitions exist so that a topic can scale beyond one machine and be consumed by many workers at once. A single ordered log can only be written and read so fast, because order is inherently sequential. Split the topic into twelve partitions and you have twelve independent logs that can be written and read in parallel, on different machines, at roughly twelve times the aggregate throughput.

The rule to carry everywhere: the partition is the unit of parallelism. Throughput scales with partition count, and, crucially, so does the maximum useful consumer parallelism. You cannot have more actively-consuming workers in a group than you have partitions, because a partition is assigned to exactly one consumer in the group at a time. Ten partitions can be shared among at most ten workers; an eleventh worker sits idle. So partition count is not an incidental setting, it is the ceiling on how much parallelism the topic can ever have, which is why it is a decision you make with the future in mind.

Keys and Ordering

Here is the constraint that partitions buy in exchange for their parallelism: Kafka guarantees order only within a partition, never across partitions.

Within one partition, records are strictly ordered and consumed in order. Across partitions, there is no ordering guarantee at all, because the partitions are independent logs read by different workers at different speeds. This matters enormously the moment order is meaningful. For our food app, the events for a single order, placed, then paid, then accepted, then picked up, must be processed in that order; processing "picked up" before "placed" is nonsense. But the events for two different orders have no required relationship and can be processed in any interleaving.

The mechanism that controls this is the key. When a producer writes a record with a key, Kafka hashes the key to choose a partition, so all records with the same key land in the same partition and are therefore strictly ordered relative to each other. Key the events by order ID, and every event for a given order goes to one partition and stays ordered, while different orders spread across all partitions for parallelism. This is the central design move in event modeling: choose the key so that things that must stay ordered share it, and things that are independent do not. Get it right and you get both ordering where you need it and parallelism everywhere else. Get it wrong, key too coarsely, say everything by region, and one hot key jams all its traffic onto one partition, destroying the parallelism and creating a bottleneck no amount of hardware fixes.

Consumer Groups: Dividing the Work

On the read side, a consumer group is how a set of workers cooperatively consume a topic. Kafka assigns each partition to exactly one consumer within the group, so the partitions are divided among the workers and each record is processed once by the group. Add workers to the group and Kafka reassigns partitions to spread the load, up to the partition-count ceiling.

The elegant part is that groups also give you fan-out for free, which is the Part 2 benefit made concrete. Different consumer groups each get their own independent view of the topic and their own offsets. The notification service is one group, analytics is another, fraud detection is a third; each reads every record at its own pace without affecting the others. So within a group you get parallelism, dividing the work, and across groups you get fan-out, everyone sees everything. Those two axes, controlled by group membership, are how a single topic serves both "spread this load across ten workers" and "let five different teams consume the same events independently."

Why Partition Count Bounds Throughput

Put the pieces together and the central capacity-planning fact appears. The maximum consuming parallelism of a topic equals its partition count, so if a single consumer can process 1,000 records per second and you have 12 partitions, the topic's ceiling is 12,000 records per second, full stop, no matter how many workers you deploy, because the thirteenth worker gets no partition and does nothing.

This makes partition count the master throughput dial, and it comes with an asymmetry worth knowing. You can add partitions to a topic later, but doing so changes how keys map to partitions, which breaks the per-key ordering guarantee for existing keys across the change, a genuinely disruptive event for ordered data. So partition count is a decision to make with headroom in mind: enough partitions to scale into your expected peak with room to spare, without going so wild that you drown in overhead, since each partition has real fixed costs in file handles, memory, and rebalancing time. A rough sizing move is to estimate peak records per second, divide by realistic single-consumer throughput, and add generous headroom. The exact number matters less than the habit of deriving it from the throughput math rather than guessing.

Rebalancing and Its Cost

When a consumer joins or leaves a group, whether from scaling, a deploy, or a crash, Kafka rebalances: it reassigns partitions among the current members. Rebalancing is necessary and it is not free, and understanding its cost prevents a class of confusing production incidents.

During a rebalance, consumption pauses while partitions are reassigned, which shows up as a latency spike and a brief backlog. Worse, a rebalance triggered by frequent scaling or by consumers that are too slow to check in, and thus wrongly presumed dead, can turn into repeated rebalancing that starves actual progress, a state where the group spends more time reshuffling than working. The practical guidance is to avoid needlessly churning group membership, size consumers so they comfortably keep up and check in on time, and treat a storm of rebalances in your metrics as the specific symptom it is rather than generic slowness. Modern Kafka has incremental rebalancing that softens the pauses, but the principle holds: membership changes have a cost, so change membership deliberately.

Delivery Semantics: At-Least-Once vs Exactly-Once

The last mechanic is the one that shapes all the correctness work in Part 4: what guarantee you have that a record is processed the right number of times. There are three possible semantics, and the middle one is where most real systems live.

• At-most-once. Each record is processed zero or one times. You never double-process, but you can lose records, a consumer reads, crashes before finishing, and the record is never retried. Acceptable only when loss is genuinely fine, which for anything touching orders or money it is not.
• At-least-once. Each record is processed one or more times. You never lose a record, but you can process it more than once, because a consumer might do its work and then crash before recording that it finished, so on restart it processes the same record again. This is the default and the pragmatic choice for most systems: nothing is ever lost, and the price is that duplicates happen.
• Exactly-once. Each record affects the outcome once, no loss and no duplication. Kafka supports this for flows that stay within Kafka, reading from a topic, processing, and writing back to a topic, through transactional guarantees. But the moment your consumer touches the outside world, charging a card, calling a third-party API, writing to a database Kafka does not control, true end-to-end exactly-once is not something the broker can give you, because it cannot make an external side effect atomic with its own offset commit.

The critical, practical conclusion is this: you should design for at-least-once and make your processing idempotent. Rather than chase an exactly-once guarantee that does not extend to your external side effects anyway, accept that duplicates will occur and ensure that processing the same event twice produces the same result as processing it once. That single design stance, embrace at-least-once, engineer for idempotency, is the foundation of the next part. Exactly-once is a real and useful tool inside pure Kafka pipelines; at the boundary where your system meets payments and databases and the customer, idempotency is what actually keeps you correct.

Conclusion

The mechanics that matter for scale all trace back to the partition. Partitions are the unit of parallelism and set the throughput ceiling; keys decide which partition a record lands in and thus what stays ordered; consumer groups divide the work up to the partition count and give independent fan-out across groups; and rebalancing is the manageable cost of changing group membership. Size partitions from the throughput math, key events so that what must stay ordered shares a key, and you have the levers that govern a Kafka deployment's scale.

Delivery semantics set up everything that follows: real systems run at-least-once, so duplicates are a fact of life, and the answer is idempotency rather than a broker guarantee that stops at the boundary of your side effects. Part 4 takes that stance and builds the full correctness toolkit on top of it: idempotency, retries, dead-letter queues, the outbox pattern, and sagas.