Lesson 18 · Distributed transactions · Module 2

The Coordination Tax

A transaction inside one database is close to free. The moment you want the same all-or-nothing outcome across two databases, you have to buy it — and the currency is a window of time in which both systems are holding locks and waiting for a third party to tell them what they did.

The win in this lesson: you will be able to explain the blocking failure mode of two-phase commit precisely enough to say why most distributed systems decline to implement it, rather than implementing it badly.

1. Two systems, one atomic outcome

The problem is easy to state and does not go away by being ignored. An order service records the order as paid in its own database. A payments service records the charge in a different database. Either both facts exist or neither should.

Inside one Postgres, this is a solved problem: one COMMIT, one write-ahead log, one durable moment where the whole transaction becomes true at once. Across two Postgres instances there is no such moment. Two commits happen at two times, and between them lies an interval in which the world is wrong. If the second commit fails, you have a charged card and an unpaid order, and nothing in either database knows about the other.

Local ACID does not compose. That is the entire subject. The rest of the lesson is what it costs to make it compose anyway.

2. The protocol

Two-phase commit splits the single commit into two round trips run by a coordinator against a set of participants. Postgres implements the participant side directly:

"PREPARE TRANSACTION prepares the current transaction for two-phase commit. After this command, the transaction is no longer associated with the current session; instead, its state is fully stored on disk, and there is a very high probability that it can be committed successfully, even if a database crash occurs before the commit is requested."

PostgreSQL 18 documentation, PREPARE TRANSACTION

Read that second clause slowly, because it is the whole trick. The transaction is no longer associated with the current session. It has outlived the connection that created it. It is a durable object sitting on disk, holding everything it acquired, with no owner — waiting to be told which way it went.

The two phases:

The coordinator's rule is simple: if every participant voted yes, send commit to all of them; if any voted no or failed to answer, send abort to all of them. Before it sends anything in phase 2 it writes its decision down, because that decision is now the only copy of the truth.

-- participant side, in each database
BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 91;
PREPARE TRANSACTION 'ord-91';   -- locks stay held, session is now free

-- ...coordinator collects the votes, writes its decision...

COMMIT PREPARED 'ord-91';       -- or ROLLBACK PREPARED 'ord-91'

Note what phase 1 does not do: it does not make anything visible. A prepared transaction is invisible to every reader and immovable by every writer that touches the same rows. It is a commit that has been paid for but not delivered.

3. What each phase costs

Phase 1 costs a network round trip and a disk flush on every participant — that is the advertised price, and it is not the interesting one. The interesting cost is the interval between the phases, and Postgres is unusually blunt about it:

"It is unwise to leave transactions in the prepared state for a long time. This will interfere with the ability of VACUUM to reclaim storage, and in extreme cases could cause the database to shut down to prevent transaction ID wraparound… Keep in mind also that the transaction continues to hold whatever locks it held. The intended usage of the feature is that a prepared transaction will normally be committed or rolled back as soon as an external transaction manager has verified that other databases are also prepared to commit."

PostgreSQL 18, PREPARE TRANSACTION · Notes

Three separate bills in one paragraph, and they escalate:

• Locks. Every row lock the transaction took is still held. Anything else touching those rows queues behind a transaction that has no session and no timeout.
• Vacuum. The prepared transaction is still an open transaction, so nothing newer than it can be reclaimed — dead tuples accumulate across the whole database, not just the touched tables.
• Wraparound. Carry that far enough and Postgres stops accepting writes entirely to protect the transaction id counter. That is the same hard stop you met in Lesson 07, arriving now for a different reason: not because vacuum was too slow, but because one forgotten transaction told it to stand still.

So the cost of 2PC is not throughput. It is that you have made somebody else's liveness into your database's correctness problem. A participant that voted yes has given up its right to decide.

4. The window where the coordinator dies

Now the failure that defines the protocol. Every participant has voted yes. The coordinator crashes before sending phase 2.

This is the property people mean when they call 2PC a blocking protocol. It is not that it is slow. It is that a single failure, in a specific window, leaves healthy participants unable to make progress and unable to be talked out of it by a timeout. There is no correct timeout to set: a participant that aborts on its own may have just diverged from one that committed, which is precisely the outcome the protocol existed to prevent.

The exit is out-of-band. Postgres exposes the stuck transactions as a view:

"The view pg_prepared_xacts displays information about transactions that are currently prepared for two-phase commit… pg_prepared_xacts contains one row per prepared transaction. An entry is removed when the transaction is committed or rolled back."

PostgreSQL 18 documentation, pg_prepared_xacts

SELECT gid, prepared, owner, database FROM pg_prepared_xacts ORDER BY prepared;
-- gid       | prepared                      | owner | database
-- ord-91    | 2026-09-23 11:04:17.221+00    | app   | orders

And the resolution can be performed by anyone with the right credentials, from anywhere: "To commit a prepared transaction, you must be either the same user that executed the transaction originally, or a superuser. But you do not have to be in the same session that executed the transaction." That sentence is what makes recovery possible at all — and it is also the admission. The protocol's recovery path is a person, or a piece of software you wrote to be that person, connecting from outside and deciding.

5. What people do instead

Almost every large system in this course reaches the same conclusion: buy a weaker guarantee that never blocks, and spend the difference on making the weakness survivable. Three moves, in rough order of how often you will meet them.

The outbox: make the cross-system write a local one

Instead of writing to two systems, write to one — your own database — twice, in a single local transaction: the business row, and a row in an outbox table describing the message to send. A separate process reads the outbox and delivers. Atomicity is now local, because both writes are in the same transaction, and delivery is now a retry problem rather than a consensus problem.

You have met this shape already. Lesson 07's migration wrote every change to an audit log inside the same transaction as the change, then replayed the log into the new shards. That is the outbox pattern doing migration work: one local commit that cannot half-succeed, and a replayer that is allowed to be slow, crash, and start again.

Sagas: a sequence of local transactions with compensations

Split the work into steps, each a complete local transaction, each with a defined compensating action. Charge the card. Mark the order paid. If step two fails permanently, run the compensation for step one — refund the charge. Nothing is ever held across a network hop, so nothing can be left in doubt.

The price is explicit and worth saying out loud: a saga gives up isolation. There is a real interval during which the card is charged and the order is not paid, and other readers can see it. With 2PC that interval was invisible but blocking; with a saga it is visible but non-blocking. You have traded a correctness property you can reason about for an availability property you can operate.

Idempotent retries: make "did that happen?" a cheap question

Both patterns above collapse into retries, and retries are only safe if repeating a step is harmless. That is Lesson 04's idempotence key earning its keep at a different layer: the outbox replayer will deliver some messages twice, the saga will re-run some steps after a crash, and neither is a bug if the downstream write is keyed so the second attempt is a no-op.

6. Residual risk

None of the alternatives are free, and pretending otherwise is how teams get hurt.

The outbox has a visible lag. The business row is committed and the message is not yet sent. If the relay stalls, your systems disagree for as long as the stall lasts, and the outbox table grows. It needs the same monitoring you would give a replication lag metric.

Compensations are not rollbacks. A refund is a new fact, not the erasure of an old one. Some steps have no compensation at all — you cannot unsend an email — so a saga forces you to order the steps so the irreversible ones come last, which is a design constraint you inherit forever.

2PC itself is not always wrong. Inside one operational boundary, with a coordinator you control and can make highly available, and with round trips measured in single-digit milliseconds, it is a legitimate tool — that is what distributed databases use internally for cross-shard writes. The judgement call is not "2PC is bad". It is: do I have a coordinator I trust more than I fear this blocking window? Across two services owned by two teams, in two regions, the answer is almost always no. Which is why the common answer is not a better 2PC, but no 2PC.

7. Check yourself

8. Back to your world

Find the place in your own system where one request writes to two things — a database and a queue, a database and a payment provider, two services behind one endpoint. Ask what happens if the process dies between the two writes. Most systems have this and have never named it; the failure shows up later as a support ticket about a record that exists on one side and not the other.

The fix is almost never 2PC. It is usually an outbox table and an idempotency key, and the conversation worth having in design review is which of the two writes is allowed to be the slow one.