Lesson 37 · Logs, offsets & consumer groups

The Log as an Abstraction

A queue forgets a message the moment someone reads it. That single property is the source of nearly every operational complaint people have about queues: you cannot replay, you cannot add a second reader cheaply, and one slow consumer drags everyone else down with it. The log deletes nothing, and hands the bookkeeping to the reader.

The win in this lesson: you will be able to choose a partition key knowing exactly which ordering guarantee you are buying — and what parallelism ceiling you have just set for every consumer group that will ever read the topic.

1. The queue that eats its own evidence

Start with what a classical broker has to do. It owns the truth about who has read what, so it must keep per-message state, forever, until that message is resolved:

"Most messaging systems keep metadata about what messages have been consumed on the broker." […] "as a message is handed out to a consumer, the broker either records that fact locally immediately or it may wait for acknowledgement from the consumer."

Apache Kafka documentation, Design — Consumer Position

Both halves of that choice are bad, and the docs are unusually candid about it. Record on hand-out and a crashed consumer loses the message. Wait for an acknowledgement and you get the other failure:

"if the consumer processes the message but fails before it can send an acknowledgement then the message will be consumed twice" […] "the broker must keep multiple states about every single message (first to lock it so it is not given out a second time, and then to mark it as permanently consumed so that it can be removed)"

Apache Kafka documentation, Design — Consumer Position

Note what that per-message lock implies. A record being processed slowly is a record the broker is holding open — memory, a timer, a redelivery decision. And the honest closing line: "Tricky problems must be dealt with, like what to do with messages that are sent but never acknowledged."

2. The log

Invert it. Do not model a topic as a mailbox that drains. Model it as a file you only ever append to, and never truncate on read.

"The log allows serial appends which always go to the last file." […] "Each message is uniquely identified by a 64-bit integer offset giving the byte position of the start of this message in the stream of all messages ever sent to that topic on that partition."

Apache Kafka documentation, Implementation — Log

Two consequences fall straight out. Records survive being read — "events are not deleted after consumption. Instead, you define for how long Kafka should retain your events through a per-topic configuration setting, after which old events will be discarded." And every record has a stable, monotonic name within its partition: the offset. A name you can write down, store, and come back to.

3. The offset is the consumer's state, not the broker's

Here is the move the whole abstraction rests on. Because records are addressable and immutable, the broker does not need to remember anything per message. It only needs each reader to say where it is:

"Our topic is divided into a set of totally ordered partitions, each of which is consumed by exactly one consumer within each subscribing consumer group at any given time. This means that the position of a consumer in each partition is just a single integer, the offset of the next message to consume. This makes the state about what has been consumed very small, just one number for each partition." […] "This makes the equivalent of message acknowledgements very cheap."

Apache Kafka documentation, Design — Consumer Position

Per-message locks, delivery counters and redelivery timers all collapse into one integer per partition per group. Reading is a pull: "The consumer specifies its offset in the log with each request and receives back a chunk of log beginning from that position."

Replay is not a feature — it is the absence of deletion

Nobody had to build rewind. It falls out of the fact that the record is still there and still has the same name:

"A consumer can deliberately rewind back to an old offset and re-consume data. This violates the common contract of a queue, but turns out to be an essential feature for many consumers."

Apache Kafka documentation, Design — Consumer Position

And a slow reader is only slow for itself

This is the property people underrate. Two groups reading the same partition share nothing except the bytes. The docs frame it as a pull-versus-push argument, and the pull side wins for exactly this reason: "a push-based system has difficulty dealing with diverse consumers as the broker controls the rate at which data is transferred", whereas "A pull-based system has the nicer property that the consumer simply falls behind and catches up when it can."

What you get back for free, in practice, is a lag number per group per partition — the gap between the log end and where that group has committed. It is the single most useful metric a log gives you, and it is arithmetic, not instrumentation:

GROUP    TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET   LAG
alerts   orders          0            9643            9643      0
alerts   orders          1            4105            4105      0
reports  orders          0            6402            9643   3241
reports  orders          1            4105            4105      0

(illustrative shape, not measured output — the point is that LAG is a subtraction)

4. Ordering is a promise about one partition — nothing wider

A topic is not one log. It is a set of logs: "a topic is spread over a number of “buckets” located on different Kafka brokers", and "When a new event is published to a topic, it is actually appended to one of the topic’s partitions."

Which one? The key decides, and that is the whole sentence you need to memorise:

"Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition’s events in exactly the same order as they were written."

Apache Kafka documentation, Introduction — Main Concepts and Terminology

Read it for what it does not say. There is no promise about two records with different keys, no promise across partitions, no global order for the topic. The producer side says the same thing from the other end: "We expose the interface for semantic partitioning by allowing the user to specify a key to partition by and using this to hash to a partition", and "For example if the key chosen was a user id then all data for a given user would be sent to the same partition."

The docs are explicit that this is a design choice made for the reader's benefit, not just a load-balancing trick: "This style of partitioning is explicitly designed to allow locality-sensitive processing in consumers." A consumer of one partition may safely keep per-key state in memory, because no other consumer in its group is touching those keys.

5. Consumer groups, and the ceiling you just set

A group is how you get parallelism without giving up the order you just bought. The assignment rule is the quote from §3, and it is doing more work than it looks: "each of which is consumed by exactly one consumer within each subscribing consumer group at any given time."

Exactly one. Not at least one. So a partition is the unit of parallelism, and the number of partitions is a hard ceiling on the number of usefully-employed consumers in any one group. Kafka 4.x makes the contrast explicit when it introduces share groups as the alternative for workloads that want queue semantics back — the first two differences listed are "The consumers in a share group cooperatively consume records, and partitions may be assigned to multiple consumers" and "The number of consumers in a share group can exceed the number of partitions in a topic."

So the partition count is a capacity decision made at topic-creation time, and it is the same decision as the ordering one. More partitions means a higher ceiling and weaker ordering; fewer means stronger ordering and a lower ceiling. There is no setting that gives you both.

6. Residual risk

Hot partitions — the same bet, in new clothes

Lesson 02's warning applies here without a single word changed. A partition key is a distribution bet, and human-generated keys follow a power law. Key by customer_id and your largest customer's traffic lands, by design, on one partition — which means one broker's disk, and exactly one consumer in each group. The parallelism ceiling is not partitions in practice; it is partitions, weighted by how evenly the key spreads.

Worse than in Lesson 02, the symptom is quiet. A hot database partition shows up as tail latency. A hot log partition shows up as lag on one partition while the topic-level average looks fine — so alert on max-partition lag, never mean lag.

Rebalances

Membership changes trigger reassignment, and the classic protocol paid for that with a synchronised stop. Kafka's newer protocol is described precisely in those terms:

"It improves the scalability of consumer groups while simplifying consumers. It also decreases rebalance times, thanks to its fully incremental design, which no longer relies on a global synchronization barrier."

Apache Kafka documentation, Consumer Rebalance Protocol

The operational reading: a rolling deploy of a consumer group is a sequence of membership changes, and anything your consumer keeps in memory per partition has to be rebuilt when that partition moves. Large local state plus frequent restarts is the combination that hurts.

Retention — replay has an expiry date

Replay is only free inside the window you paid for. Past it, the record is gone and the offset you stored is meaningless: "when the client attempts to consume a non-existent offset it is given an OutOfRangeException". A consumer that has been down longer than the retention period does not resume — it jumps, silently, to whatever its reset policy says, and skips everything in between.

7. Check yourself

8. Back to your world

You do not need a streaming platform to have made these decisions. Any time you put work on a queue keyed by tenant, fan out events to more than one downstream, or find yourself writing we should be able to re-run yesterday's events, you are choosing between a mailbox and a log — usually without saying so.

Three questions to carry out: which two records in this stream must never be reordered, and do they share a key? How many consumers could I usefully run, and did I set that number by accident? And how long is replay actually true for — a day, a week, or until the next deploy?