Lesson 36 · The outbox pattern · Module 5

The Outbox

Save the row, publish the event. Two writes, two systems, one request — and nothing whatsoever holding them together. Sooner or later the first succeeds and the second does not, and the two stores disagree for ever without anybody being told.

The win in this lesson: you will spot a dual write in any design on sight, you will know the two-line fix, and you will know exactly what that fix costs — publishing becomes asynchronous, so everything downstream is now eventually consistent.

1. The dual write

The code is so ordinary that it reads as correct:

order = orders.insert(...)        # database
queue.publish("OrderCreated", …)  # broker
return 201

Two network calls to two systems that have never heard of each other. There is no transaction spanning them, no shared commit record, nothing that fails both together. So watch what happens when the second one does not answer.

Swap the order and you swap the bug, not the risk. Publish first and the commit may fail: now a consumer is acting on an order the database has never heard of. That one is worse, because it is contagious — downstream systems build state on an event that describes nothing.

2. Why you cannot make it atomic

Every arrangement of two writes to two systems has the same hole in it. There is no ordering that closes it, because the hole is the gap between the two calls, and that gap always exists.

The fourth row is the one people reach for, and it is why Lesson 18 matters here. Two-phase commit really does give you the all-or-nothing outcome. It also makes both participants surrender the right to decide alone, so a coordinator that dies between phases leaves them holding locks with no legal move. Most brokers cannot join such a protocol anyway.

3. The outbox

"The outbox pattern is a way to safely and reliably exchange data between multiple (micro) services. An outbox pattern implementation avoids inconsistencies between a service’s internal state (as typically persisted in its database) and state in events consumed by services that need the same data."

Debezium, Outbox Event Router

The outbox is an ordinary table in your ordinary database. Debezium's default expects these columns:

"To apply the default outbox event router SMT configuration, your outbox table is assumed to have the following columns:"

Debezium, Outbox Event Router, "Basic outbox table"

Column        |          Type          | Modifiers
--------------+------------------------+-----------
id            | uuid                   | not null
aggregatetype | character varying(255) | not null
aggregateid   | character varying(255) | not null
type          | character varying(255) | not null
payload       | jsonb                  |

And the fix to the request path is two lines — one added, one deleted:

BEGIN;

INSERT INTO orders (id, customer_id, total)
VALUES ('1', 123, 69.97);

INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)   -- added
VALUES (gen_random_uuid(), 'orders', '1', 'OrderCreated',
        '{"id": 1, "customerId": 123}');

COMMIT;            -- both rows, or neither. there is no third outcome.

-- queue.publish(...)                                               -- deleted

That is the whole trick. The event is no longer a message sent over a network that may or may not arrive; it is a row, protected by the same commit as the state change it describes. The database's atomicity, which you were already paying for, now covers both facts.

4. The relay, and where you have seen it before

A committed outbox row still has to reach the broker. A separate process reads the rows and publishes them — and the good version does not query the table at all. It reads the log the database is already writing:

"The MySQL connector uses a client library for accessing the binlog[…] The PostgreSQL connector reads from a logical replication stream."

Debezium, Debezium Architecture

Which should feel extremely familiar. Lesson 01's mcsqueal tailed the replication stream to invalidate caches. Lesson 07's migration wrote an audit log row inside the request and had a catch-up process replay it — explicitly not writing to both stores inline, for exactly the reason this lesson opens with. The outbox is that same move, generalised: record the intent durably in one commit, deliver it asynchronously. Lesson 07 replayed into a second database; here you replay into a broker.

Debezium hands you the deduplication key in the message itself: the id column "Contains the unique ID of the event. In an outbox message, this value is a header. You can use this ID, for example, to remove duplicate messages."

5. What it costs

The outbox does not remove failure. It moves failure to the one place where retrying is harmless, and converts an invisible divergence into a visible delay. That trade has a bill.

Duplicates are now certain, not possible

The relay publishes, then records its position. A crash between those two acts republishes. This is at-least-once delivery — Lesson 35's subject — and it is not a defect you can configure away. Consumers must be idempotent, keyed on the event id.

Ordering is commit order, per key — and nothing more

The rest of the bill

6. Check yourself

7. Back to your world

Open any service you own and search the request path for a database write and a network publish in the same function. Email, webhook, analytics event, cache invalidation, search index update, another team's API — they are all the same bug wearing different clothes. Then ask the three questions that decide the fix:

• Which write is the source of truth? That one stays in the transaction. The other becomes an outbox row.
• Can the consumer survive seeing it twice? If not, you have not finished. Delivery is at-least-once and no setting changes that.
• Who deletes the old rows? An outbox nobody prunes becomes the largest table in your database, and you will find out during an incident.