Reference · quick sheet

The Staleness Playbook

Replicas, lag, and the four ways to make a user see their own write. Print it; it is designed to survive a black-and-white printer.

Vocabulary

Replication

A replica follows the primary's durable change log (WAL / binary log) and replays every entry in order into its own full copy. Log following, not a copy job — and not a backup, since it replays your mistakes too. Makes copies; does not split data (that is partitioning).

Streaming replication

The standby holds a long-lived connection to the primary (walsender → walreceiver) and WAL records flow down it as they are generated — not once a 16 MB segment fills, which is the older file-based log shipping. Postgres docs: typically under one second, if the standby keeps up.

Replication handshake

Standby dials out (replication=true → walsender mode), asks IDENTIFY_SYSTEM, then START_REPLICATION SLOT … LSN naming its own starting position — once. After CopyBothResponse the primary pushes 'w' WAL messages unasked; the standby replies 'r' with its written / flushed / applied LSNs. Pull to open, push to deliver.

Replication slot

The primary tracking how far a standby has consumed, so it will not recycle WAL still needed. Cost: an offline standby's slot can fill the primary's disk.

LSN

Log sequence number — a byte position in the WAL. The primary has a head position; the standby has a received position and a replayed position. Only replayed is visible to queries there.

Replication lag

The distance between the primary's head LSN and the standby's replayed LSN. A long-tailed distribution, not a constant, and worst under write bursts.

Read-your-writes

Guarantee: after a user writes, that same user's later reads reflect it. Other users may lag. Also called read-after-write.

Monotonic reads

Guarantee: a user never sees time run backwards across successive reads. Typically broken by bouncing a user between replicas of differing freshness.

synchronous_commit

Postgres' dial for who waits for whom: off (no wait) → local (local flush only) → remote_write (standby wrote it) → on, the default (standby flushed it durably) → remote_apply (standby applied it — visible to queries there). remote_apply is rung 4. Each level waits on one field of the standby's 'r' status message — same stream, different patience.

Change data capture (CDC)

Reading a database's commit log and emitting the committed changes as events for other systems. Facebook's mcsqueal is CDC with one consumer: it extracts invalidations embedded in committed SQL and broadcasts them to every frontend cluster in its region. Because the events live in the log, a lost one can be replayed. Same shape as Debezium, Postgres logical decoding, DynamoDB Streams.

Frontend cluster / storage cluster

Facebook's two cluster kinds inside a region. Frontend = web servers + memcached + mcrouter, holding nothing that cannot be rebuilt; disposable, randomly routed to. Storage = MySQL + mcsqueal, the only truth. "Frontend" is a tier word, not a browser word — the cache belongs to it, which is why cached copies multiply per cluster and invalidation is broadcast outward from storage.

Look-aside cache

The app asks the cache first and, on a miss, reads the database (or another backend service) and populates the cache itself. The cache is not in the write path and is never authoritative — which is what makes eviction always safe.

Idempotent

Safe to apply more than once with the same result. Why cache invalidation is a delete, not an update — deletes survive retries and reordering.

k / r_k

k is a memcache key — the string an app invents to name a cached value (user:1234:profile), not a row id. r_k is the remote marker belonging to that key: one per key, not one per region.

Remote marker

Facebook's per-key flag meaning "this region's copy may be behind." Its presence is the whole message — the value is irrelevant. Present ⇒ route the read to the master region. Stored in the regional pool so every cluster in the region sees it.

The ladder — pick the lowest rung that clears the bar

Measuring it (PostgreSQL 18)

SELECT pg_current_wal_lsn();          -- primary: head of the log
SELECT pg_last_wal_receive_lsn(),     -- standby: received and flushed
       pg_last_wal_replay_lsn();      -- standby: replayed — queries see this

SELECT pg_wal_lsn_diff(               -- lag in bytes
         '3/AF0001C8', pg_last_wal_replay_lsn());
SELECT now() - pg_last_xact_replay_timestamp();   -- lag in time

Design-review questions

1. After this write, what is the very next read, and which node serves it?
2. Is the stale value merely displayed, or is it cached? Caching a stale read freezes the mistake in place for everyone.
3. Is the stale value used to compute the next write? That upgrades a cosmetic bug into data corruption.
4. What is lag at p99, not on average — and what does the product do during the bad tail?
5. Which users need the guarantee? Usually "the author of the write", almost never "everyone".
6. Where does the state that enforces this live — in the infrastructure, keyed by data, or on the client, keyed by the user? What happens when the user isn't where it is?
7. If the replica is 10 minutes behind, does the system get slow or get wrong? Slow is survivable.

Say the price out loud

We explicitly trade ⟨cost⟩
for ⟨reduction in badness⟩,
and it still fails when ⟨case⟩,
which we accept because ⟨evidence⟩.

Modelled on NSDI '13 §5: "we explicitly trade additional latency when there is a cache miss, for a decreased probability of reading stale data." Note the honesty of "probability".

Who did what