Lesson 10 · Storage engines · Module 1
The Three Amplifications
Two lessons ago a B-tree made reads cheap and taxed every write. Last lesson an LSM-tree made writes cheap and taxed every read. That is not two unrelated facts. It is one budget, spent in different places — and there are exactly three places to spend it.
The win in this lesson: given a workload, you will be able to say which of the three amplifications you should be willing to pay, and name the knob that trades it for another. Not "it depends" — a corner, and a setting.
1. The problem: three bills, one engine
Arguments about storage engines go round in circles because the participants are measuring different things and nobody says which. One person means "my p99 read got slower". Another means "the disks are at 80% write utilisation". A third means "we are paying for 3 TB to store 900 GB of data". All three are real, all three are consequences of the same design decision, and moving any one of them moves the others.
RocksDB — Facebook's LSM storage engine, the thing under MyRocks, CockroachDB and a long list of others — is unusually blunt about this, because its tuning guide is essentially a document about how to choose which bill to pay. It opens by naming the three:
"Write amplification is the ratio of bytes written to storage versus bytes written to the database."
"Read amplification is the number of disk reads per query."
"Space amplification is the ratio of the size of database files on disk to data size."
RocksDB Tuning Guide, Amplification factors
And then, one line later, the sentence this whole lesson hangs on:
"Either way, compaction is key to change the trade-off among the three."
RocksDB Tuning Guide, Amplification factors
Change the trade-off, not remove it. Compaction is the steering wheel, not the brake.
2. The three quantities, precisely
Each one is a ratio or a count with a definite denominator. Say the denominator out loud when you use the term, and the circular arguments stop.
Write amplification
You insert 1 KB. How many kilobytes hit the device? For an LSM-tree the answer is not 1, because the same 1 KB is written once on flush and then rewritten every time compaction merges it into a bigger level. For a B-tree the answer is not 1 either, because the device writes whole pages: a 40-byte column update rewrites an 8 KB page, plus a page in every index on the table, plus the WAL record.
This is the number your SSD warranty is denominated in. Write amplification is not a latency problem first — it is a device lifetime and device bandwidth problem. A flash drive rated for three drive writes per day at amplification 1 is rated for a tenth of a drive write per day at amplification 30.
Read amplification
You ask for one key. How many disk reads happen? For a B-tree it is the depth of the tree, minus whatever is cached — a handful, and stable, which is the whole argument of Lesson 08. For an LSM-tree it is one read per sorted run that might contain the key, which is a number that grows as you let runs accumulate.
Space amplification
Your live data is 900 GB. How many gigabytes are on disk? Above 1 because of obsolete versions not yet compacted away, tombstones not yet dropped, and — in a B-tree — pages deliberately left part-empty so that the next insert does not split them. This is the amplification people forget in design review, and then meet as a capacity incident six months later.
"Write amplification of 30" means 30 bytes to the device per byte you handed the database. "Read amplification of 4" means four disk reads per query. "Space amplification of 1.11" means 11% overhead on top of live data. If someone in a design review says "this is more efficient", the useful question is not "how much" — it is "in which of the three?", because efficiency in one is almost always purchased from another.
3. Why you cannot have all three
The reason is not engineering immaturity. It is that the three quantities are bound together by what storage physically is.
To make reads cheap you need the data sorted and in few places, so a lookup goes straight to it. To make writes cheap you must stop sorting things on the write path — take the data as it arrives, append it, sort later. But "sort later" means the data temporarily lives in many places at once, which is read amplification, and it means old copies coexist with new ones, which is space amplification. The only way to collapse those copies back down is to rewrite them — which is write amplification, the thing you were trying to avoid.
Push down on any one corner and the other two rise. This is the shape that the academic literature calls the RUM conjecture — Read overhead, Update overhead, Memory (space) overhead — proposed by Athanassoulis and colleagues in 2016: an access method can be designed to be excellent in two of the three, never all three at once. It is a conjecture rather than a theorem, but no one has produced the counterexample, and RocksDB's own documentation is written as if it were settled law.
4. How real engines pick a corner
RocksDB is a good place to watch this happen because it ships two compaction strategies with opposite answers, and documents why.
Leveled compaction: pay in writes
The default. Data is organised in levels, each an order of magnitude bigger than the one above:
"Files on disk are organized in multiple levels. We call them level-1, level-2, etc"… "Each level (except level 0) is one data sorted run".
RocksDB Leveled Compaction, Structure of the files
"Each level is 10 times larger than the previous one (this multiplier is configurable)", says the tuning guide. Two consequences follow immediately, and they are the two halves of the trade.
Read amplification is low, because one level is one sorted run, so there is at most one file per level to consider, and a bloom filter usually eliminates even that. Space amplification is low and, unusually, bounded — the geometric progression means the bottom level dominates:
"We can guarantee 90% of data is stored in the last level, 9% data in the second last level".
RocksDB Leveled Compaction, Guaranteed Space Amp Upper Bound
You pay for both in the third currency. Every byte is rewritten on its way down through every level, and with a 10× multiplier the merge at each step reads roughly ten bytes of destination per byte of source:
flush memtable to L0 ................. 1x
L0 merged into L1 .................... 2x
L1 merged into L2 .................... 10x
L2 merged into L3 .................... 10x
L3 merged into L4 .................... 10x
----
33x"Total write amplification is therefore approximately
RocksDB Tuning Guide, Level Style Compaction1 + 2 + 10 + 10 + 10 = 33."
Thirty-three bytes to the device for every byte you inserted. That is the price of the cheap reads and the bounded disk footprint, stated in the vendor's own documentation, and it is a price a great many production systems happily pay.
Universal compaction: pay in reads and space
When that bill is too big, RocksDB offers the opposite corner. It exists for exactly one reason, and the wiki says so in its first sentence:
"Universal Compaction Style is a compaction style, targeting the use cases requiring lower write amplification, trading off read amplification and space amplification."
RocksDB Universal Compaction, Conceptual Basis
Instead of levels, sorted runs by age: "all the SST files are organized as sorted runs covering the whole key ranges", and "Compaction can only happen among two or more sorted runs of adjacent time ranges." Merging only adjacent, similarly-sized runs means each byte is rewritten far fewer times — "It is generally regarded that the second strategy provides far better write amplification with worse read amplification."
The worse read amplification is structural: more sorted runs coexist, so more of them must be consulted. And the space cost has a sharp edge that catches people out:
"In universal style compaction, sometimes full compaction is needed. In this case, output data size is similar to input size."… "During compaction, both of input files and the output file need to be kept, so the DB will be temporarily double the disk space usage."
RocksDB Universal Compaction, Limitations · Double Size Issue
Read that as a capacity-planning rule, not a footnote: a universal-compaction store can need 2× its steady-state footprint, transiently, at a moment you do not control. Plan the volume for the peak or the peak arrives at 3 a.m.
The knobs
Every one of these moves the budget from one corner to another. None of them reduces the total.
| Knob | Turn it to… | You buy | You pay |
|---|---|---|---|
compaction_style |
universal instead of level | much lower write amplification | higher read and space amplification |
max_bytes_for_level_multiplier |
larger (more than 10) | fewer levels, so lower read amplification | more rewriting per level — write amplification |
max_size_amplification_percent |
lower | a tighter cap on space amplification | more full compactions — write amplification |
| bloom filter bits per key | higher | fewer wasted disk reads per point lookup | RAM, and nothing else — the honest exception |
| compression on the bottom level | stronger | lower space amplification | CPU on every read, and slower compaction |
| B-tree fill factor | lower (leave pages emptier) | fewer page splits — write amplification | a bigger index on disk — space amplification |
The max_size_amplification_percent row is worth dwelling on, because it is the trade made
explicit in a single setting: "If the estimated size amplification ratio is larger than
options.compaction_options_universal.max_size_amplification_percent / 100, all files will be
compacted to one sorted run." You declare how much space amplification you will tolerate, and the engine
spends write amplification to hold you to it. That is the whole lesson in one config line.
5. Choosing your corner
The decision procedure is short. Find the resource you are closest to exhausting, and pay in the other two.
| If your workload looks like… | Be willing to pay… | Because… |
|---|---|---|
| Huge ingest, rare reads — metrics, logs, event capture | read and space | device write bandwidth is the binding constraint; nobody is waiting on a read |
| Read-dominated with a strict p99 — a user-facing lookup path | write and space | a latency SLO is a promise; disk is cheap and rewriting happens in the background |
| Enormous dataset, cost-dominated — archival, cold storage | read and write | the bill is proportional to bytes stored, so space amplification is the bill |
| Flash you must not wear out — dense SSDs, long refresh cycle | read and space | write amplification is denominated in drive lifetime, which you cannot buy back later |
6. Residual risk
Three ways this bites even when the corner was chosen correctly.
The averages are fine and the tail is not. Amplification figures are steady-state averages. Compaction is bursty, and universal compaction is burstier by design — the wiki notes that "The lazy nature of the compaction scheduling also makes the compaction traffic much more spiky, the number of sorted runs greatly varies over time, hence large variation of performance." Your p99 read is served during a compaction storm, not between them. This is Lesson 02's tail-latency lesson arriving from underneath.
The workload moves and the corner does not. The engine was tuned for the shape of the traffic at the time. Add a reporting query, a backfill, a new access pattern that does range scans over what used to be point lookups, and you are standing in the wrong corner with a configuration nobody remembers choosing.
Space amplification fails discontinuously. Read and write amplification degrade gracefully — things get slower. Space amplification does not: the volume is 100% full and writes stop. That is the same failure mode as Lesson 07's transaction-id wraparound — a cliff rather than a slope — and it deserves the same treatment: a monitor on the ratio, alerting long before the disk-usage graph looks alarming.
7. Check yourself
8. Back to your world
Pick the busiest table in your main database and work out all three numbers, roughly. Bytes written to
the device per hour, divided by bytes of application data inserted per hour. Pages read per query from
EXPLAIN (ANALYZE, BUFFERS). Total relation plus index size, divided by an estimate of live row
bytes. You will almost certainly find one of the three is an order of magnitude worse than you assumed —
and that is the one your next design decision should be spending, or saving.
Then ask the question that makes this a tech-lead skill rather than a trivia one: when someone proposes a new index, a new column, a new retention policy, which corner does it push on? Every proposal moves the budget. Most design reviews only notice when it moves the corner somebody is currently standing in.