Lesson 35 · Asynchronous work · Module 5

At Least Once, At Most Once, and the Exactly-Once Myth

A sender that never hears back cannot tell "it did not arrive" from "it arrived and the reply was lost". Those two worlds look identical on the wire, and they demand opposite actions. Every delivery guarantee you have ever configured is a decision about which of them you would rather be wrong about.

The win in this lesson: you will be able to look at any producer/consumer pair and say which semantics it really has — not which it advertises — and point to the exact place where deduplication would have to live before anyone is allowed to say "exactly once".

1. The problem: silence is ambiguous

A producer sends a record and waits. The acknowledgement does not come. The send may have failed on the way out, or the record may be durably written and the acknowledgement lost on the way back. Kafka's design document states the ambiguity as flatly as it can be stated:

"If a producer attempts to publish a message and experiences a network error, it cannot be sure if this error happened before or after the message was committed. This is similar to the semantics of inserting into a database table with an autogenerated key."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

The whole subject is in the shaded band. A timeout is not evidence of failure — it is the absence of evidence, and the sender must act anyway. Resend and you may duplicate; stay quiet and you may lose. Every delivery guarantee below is a name for one of those two answers.

The autogenerated-key analogy is worth sitting with. You INSERT, the connection drops, and you never learn the generated id. Retry and you may create a second row; do not retry and you may have created none. The queue has not invented a new problem — it has made a database problem unavoidable, because a queue's whole job is to be a network hop.

2. The three semantics, and which two are real

The classification is three lines long, and the source gives it without ceremony:

"At most once […] Messages may be lost but are never redelivered. At least once […] Messages are never lost but may be redelivered. Exactly once […] Each message is processed once and only once."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

Read the third definition again. It says processed, not delivered. That is not sloppy wording; it is the whole escape route, and section 4 is about walking down it.

Semantics	What the sender does on silence	What you must build	Failure you accept
At most once	Gives up. Retries disabled, offset committed before work	Nothing	Silent loss, invisible in metrics
At least once	Resends until acknowledged	Bounded retries and backoff	Duplicates, at any multiplicity
Exactly once	Resends, and is deduplicated on arrival	A key the sender repeats, and receiver-side state that remembers it	The scope of that state — per what, for how long

Only the first two are wire behaviours. The third is a property of the effect, reconstructed above a wire that is still doing one of the first two. The documentation is unusually direct about the marketing:

"Many systems claim to provide "exactly-once" delivery semantics, but it is important to read the fine print, because sometimes these claims are misleading […]"
Apache Kafka documentation · Design · §"Message Delivery Semantics"

3. Where the acknowledgement boundary sits

"Acknowledged" is not a fact about the network; it is a line someone drew, and you choose where. On the producer side the strongest line is durability across the replica set:

"A message is considered committed only when all replicas in the in-sync replicas (ISR) for that partition have applied it to their log."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

You may move that line inwards for latency — wait only for the leader, or do not wait at all. Each step inwards widens the set of failures that can eat an acknowledged message. Moving the line does not remove the ambiguity of silence; it changes what an acknowledgement was worth when you did get one.

The problem then splits cleanly in two, and most teams only think about the first half:

"It's worth noting that this breaks down into two problems: the durability guarantees for publishing a message and the guarantees when consuming a message."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

4. The consumer half: commit order is your semantics

A consumer holds two things that can be saved: its position in the log, and the output of processing. Nothing makes them atomic by default, so one of them is written first — and that ordering, a single line of code, silently decides your delivery guarantee.

"It can read the messages, then save its position in the log, and finally process the messages. […] This corresponds to "at-most-once" semantics as in the case of a consumer failure messages may not be processed."

"It can read the messages, process the messages, and finally save its position. […] This corresponds to the "at-least-once" semantics in the case of consumer failure."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

Nobody configures at-most-once. They inherit it — from auto-commit on a timer, or from a loop that acknowledges on receipt because that felt tidy. If you cannot say which of these two diagrams your consumer draws, you do not know whether your system loses work or repeats it.

# A — at-most-once: the position moves first
records = consumer.poll()
consumer.commit_sync()        # crash after this line: work never happens
process(records)

# B — at-least-once: the position moves last
records = consumer.poll()
process(records)              # crash after this line: work happens twice
consumer.commit_sync()

The defaults are not neutral either. Kafka states which way it leans, and what you must switch off to get the other:

"Otherwise, Kafka guarantees at-least-once delivery by default, and allows the user to implement at-most-once delivery by disabling retries on the producer and committing offsets in the consumer prior to processing a batch of messages."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

5. Where "exactly once" actually comes from

Not from the network. It is manufactured at the receiving end, in one of two ways, and both are worth being able to name.

5.1 The receiver deduplicates

The sender keeps resending — that part never changes — and the receiver recognises a repeat and declines to apply it twice. Kafka's idempotent producer is exactly this, and the mechanism is small:

"Since 0.11.0.0, the Kafka producer also supports an idempotent delivery option which guarantees that resending will not result in duplicate entries in the log. To achieve this, the broker assigns each producer an ID and deduplicates messages using a sequence number that is sent by the producer along with every message."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

This is the shape of every exactly-once claim ever made: a stable identifier the sender repeats, and state at the receiver that remembers it. Change the identifier on retry and the mechanism evaporates — which is why an idempotency key must be chosen by the caller, before the first attempt, not generated per request.

5.2 The effect is idempotent anyway

The cheaper route, and the one most systems should take. If applying the same message twice leaves the same state, duplicates stop being errors:

"In many cases messages have a primary key and so the updates are idempotent (receiving the same message twice just overwrites a record with another copy of itself)."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

5.3 Making the offset and the output atomic

The consumer's dilemma in section 4 exists only because two writes are not one write. Kafka's answer is to make them one, by putting the offset inside the same transaction as the output:

"The consumer's position is stored as a message in an internal topic, so we can write the offset to Kafka in the same transaction as the output topics receiving the processed data."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

The division of labour surprises people, so note which component is transactional:

"[…] it is only the producer which is transactional. It is however able to make transactional updates to the consumer's position (confusingly called the "committed offset"), and it is this which gives the overall exactly-once behavior."
Apache Kafka documentation · Design · §"Using Transactions"

# exactly-once processing, Kafka-to-Kafka: one atomic write, or none
producer.begin_transaction()
producer.send(output_records)
producer.send_offsets_to_transaction(offsets, group_metadata)
producer.commit_transaction()

# the configuration that makes it mean anything
isolation.level    = read_committed   # do not read aborted work
enable.auto.commit = false            # the offset moves in the transaction, not on a timer
transactional.id   = <stable per instance>

Two details make or break it. The reader must use read_committed, or it "will see records from aborted transactions and also open transactions which have not yet completed". And the consumer group must guarantee single ownership — the source's first requirement is that "The consumer uses partition assignment to ensure that it is the only consumer in the consumer group currently processing each partition.".

The one idea

Exactly-once is never a property of delivery; it is a property of application. The wire will always be at-most-once or at-least-once, because silence is ambiguous and always will be. So when somebody claims exactly-once, ask one question: where is the deduplication? Name the component, the key it deduplicates on, and how long it remembers. If there is no answer to all three, the system is at-least-once with better branding — and that is fine, provided the effects are idempotent.

6. Residual risk

The guarantee ends at the edge of the system that holds the deduplication state. Cross that edge and you are back in section 1:

"When writing to an external system, the limitation is in the need to coordinate the consumer's position with what is actually stored as output."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

The classic fix is two-phase commit, and the classic advice is to avoid it — most sinks cannot play, and the blocking window is real. The recommended move is simpler and, once seen, obvious:

"This can be handled more simply and generally by letting the consumer store its offset in the same place as its output."
Apache Kafka documentation · Design · §"Message Delivery Semantics"

Write the offset into the destination database, in the same transaction as the rows. One atomic write replaces a distributed agreement — the same trick as 5.3, aimed at a different store. And for everything else, the documentation is honest about the limit:

"Exactly-once delivery for other destination systems generally requires cooperation with such systems, but Kafka provides the primitives which makes implementing this feasible […]"
Apache Kafka documentation · Design · §"Message Delivery Semantics"

Three risks survive every configuration on this page:

Effects that leave the transaction. An email, a payment call, a push notification. No rollback exists for those, so they must carry their own idempotency key at the far end.
Deduplication state is finite. Broker-side sequence numbers are per producer session and per partition; an application dedup table has a retention window. A retry that arrives after the memory expires is a duplicate again, and a reprocessing of last month's log is well past every window.
At-most-once is usually accidental. Nobody asks for loss. They get it from auto-commit on a timer, or from acknowledging on receipt, and the evidence is missing records rather than an error anybody can alert on.

7. Check yourself

8. Back to your world

Pick one consumer you own and answer three questions about it, in order. When does it acknowledge or commit — before the work, or after? That single fact tells you whether your system loses records or repeats them, and it is usually a default nobody chose. If it repeats them, what happens on the second pass? An upsert keyed by a business id is a non-event; an increment, an append or an outbound email is an incident. And if someone has told you the pipeline is exactly-once, which component holds the dedup state, on what key, for how long?

If the third question has no owner, you are running at-least-once. Stop trying to fix the wire and make the effect idempotent instead — it is cheaper, it survives replays, and it keeps working the day somebody reprocesses the last month of the log to fix a bug.

Ask me things. "walk me through an idempotency key for a payments endpoint" · "what exactly does a transactional producer write to disk?" · "how big does a dedup table get, and how do I expire it?" · "show me the offset-in-the-sink pattern as code" · "I think our pipeline is genuinely exactly-once. Grill me."