System Design.

Reference · quick sheet

The Migration Playbook

The same four phases whether you are sharding a database, changing a column type, or replacing a queue. Load phases 1–3 until phase 4 has almost nothing to do.

The four phases

  1. Double write. Every write goes to both stores. Do it through a durable audit log plus a catch-up process, not two inline writes — two inline writes are two independent chances to fail and a silent divergence when the second one does. A logged write is retryable, restartable and observably behind.
  2. Backfill. Copy everything written before phase 1. Notion's took about three days. Checkpoint it so a crash resumes rather than restarts.
  3. Verify with dark reads. Read both, serve the old one, compare in the background. Every real request becomes a test at production scale with no user risk. This is the phase teams skip and the one that turns "we believe it is correct" into a number.
  4. Flip. Reads move over; the old store retires. The only irreversible step, and the only one users can see.

Rules

  • Phases 1–3 can run for weeks, be paused, and be abandoned with nothing lost but time. Only phase 4 is one-way. Keep it that way.
  • If phase 4 is doing something you have not rehearsed, move that work earlier.
  • Every step must be safe to run twice. Idempotence is what makes a stalled migration resumable.
  • Know your rollback at each phase, and confirm it still works after the backfill, not before.
  • Notion took five minutes of downtime and afterwards judged that "another week optimizing the script" would have removed even that. Decide deliberately which you are spending: a week of engineering, or minutes of downtime.

Sharding specifics

The unit is a subgraph, not a table
Notion sharded "all tables reachable from the block table via some kind of foreign key relationship". Leave a related table unsharded and any join to it crosses machines.
Decouple logical from physical
480 logical shards on 32 machines. 480 is highly composite, so growth is 32 → 40 → 48 — redistribution, never re-sharding.
Key ergonomics are permanent
Their own regret: carrying both id and space_id when "we could've combined both keys into a single new column".

The deadline that is not about performance

Postgres stamps rows with a 32-bit transaction id and VACUUM must freeze old rows before the counter laps them. If VACUUM stalls, Postgres refuses all writes rather than risk corruption. Not a slowdown — a hard stop, scheduled by your write rate, unbuyable with hardware. Watch for stalling vacuum long before you watch for slow queries.