Lesson 20 · Time, order and truth · Module 3

You Cannot Trust a Timestamp

Every machine in your system is confidently telling you what time it is, and every one of them is wrong by a different amount. That is survivable right up until you use those numbers to decide which of two writes happened later — at which point the system starts deleting data and reporting success.

The win in this lesson: you will never again use a wall-clock timestamp to order two events on different machines, and you will know exactly which clock to read when you want to measure how long something took.

1. The failure, first

Two nodes accept writes to the same key. Each stamps its write with its own clock. The store keeps the write with the higher stamp — last write wins. Nothing here is exotic; it is the default in more systems than you would like.

Here is what that does when the two clocks disagree by a quarter of a second. Time runs downward, and downward is true time — the order things actually happened in:

2. What is actually inside the clock

Every computer keeps time by counting the vibrations of a small piece of quartz. That is a manufacturing process, so it has a tolerance:

"Quartz clocks are cheap, but they are not totally accurate. Due to manufacturing imperfections, some clocks run slightly faster than others. Moreover, the oscillation frequency varies with the temperature."

M. Kleppmann, Concurrent and Distributed Systems, §3.1 Physical clocks, Slide 45

The rate of that error is called drift, measured in parts per million. The notes put a number on it that is worth memorising, because it converts an abstract worry into a budget:

"1 ppm = 1 microsecond/second = 86 ms/day = 32 s/year […] Most computer clocks correct within ≈50 ppm"

Kleppmann, §3.1, Slide 45

Fifty parts per million is 4.3 seconds of error per day, per machine, in opposite directions on two different machines. Sitting an unsynchronised node next to a hot exhaust makes it worse, because "significantly higher or lower temperatures slow down the clock". This is why the 280 ms gap in the diagram above is not a contrived number — it is what an hour of unsynchronised running buys you, and the whole business of NTP exists to stop it growing.

3. What NTP does to you

NTP is the fix for drift, and it is also the second reason you cannot trust a timestamp. The client estimates the skew between its clock and a server's, then corrects it — and how it corrects depends on how big the error is:

Thresholds from Kleppmann, §3.2, Slide 56.

The middle row is the one that hurts:

"[…] if the skew is larger, slewing would take too long, so the NTP client instead forcibly sets its clock to the estimated correct time based on the server timestamp. This is called stepping the clock. Any applications that are watching the clock on the client will see time suddenly jump forwards or backwards."

Kleppmann, §3.2 Clock synchronisation and monotonic clocks

And the bottom row of the table is worse than it looks, because it is silent:

"[…] any system that depends on clock synchronisation needs to be carefully monitored for clock skew: just because a node is running NTP, that does not guarantee that its […] clock will be correct, since it could get stuck in a panic state in which it refuses to adjust the clock."

Kleppmann, §3.2

"Is NTP running?" is therefore not the question. The question is "what is the measured skew on this node right now", and if you are not exporting that as a metric, you do not know.

4. The two clocks, and which one you meant

Every operating system gives you two different clocks behind two different calls, and almost every elapsed-time bug is someone reaching for the wrong one.

Rows from Kleppmann, §3.2, Slide 59.

"[…] if you use such a clock to measure elapsed time, the resulting difference between end timestamp and start timestamp may be much greater than the actual elapsed time (if the clock was stepped forwards), or it may even be negative (if the clock was stepped backwards). This type of clock is therefore not suitable for measuring elapsed time."

Kleppmann, §3.2, on the time-of-day clock

// wrong: two readings of a clock that can be reset between them
start   = clock_gettime(CLOCK_REALTIME)
work()
elapsed = clock_gettime(CLOCK_REALTIME) - start   // may be negative

// right: two readings of a counter that only moves forwards
start   = clock_gettime(CLOCK_MONOTONIC)
work()
elapsed = clock_gettime(CLOCK_MONOTONIC) - start  // always >= 0

But the monotonic clock is not a licence to compare across machines. It buys you safety by giving up meaning:

"[…] a timestamp from a monotonic clock is meaningless by itself: it measures the time since some arbitrary reference point, such as the time since this computer was started up. When using a monotonic clock, only the difference between two timestamps from the same node is meaningful. It does not make sense to compare monotonic clock timestamps across different nodes."

Kleppmann, §3.2

5. Why last-write-wins is a deletion policy

Now put §1 and §3 together. The ordering failure in the first diagram is not bad luck; it is the guaranteed consequence of a residual uncertainty that synchronisation cannot remove:

"The clock synchronisation performed by NTP and similar protocols always leaves some residual uncertainty about the exact skew between two clocks, especially if the network latency in the two directions is asymmetric. […] if we order messages based on their timestamps from time-of-day clocks, we might again end up with the wrong order."

Kleppmann, §3.3 Causality and happens-before, Slide 61

The vocabulary for what you actually wanted is Jepsen's. The strongest single-object model is the one that promises exactly the property a timestamp appears to offer and does not deliver:

"Linearizability is one of the strongest single-object consistency models, and implies that every operation appears to take place atomically, in some order, consistent with the real-time ordering of those operations: e.g., if operation A completes before operation B begins, then B should logically take effect after A."

Jepsen, Linearizability

Note what that costs: linearizability is a property a system implements, with coordination, and "in the event of a network partition, some or all nodes will be unable to make progress". Last-write-wins is the opposite trade — it keeps writing during a partition and pays for it in silently lost updates. That is Lesson 05's dial again: last-write-wins was Dynamo's cheapest conflict resolution, and it is a data-loss policy wearing a neutral name.

So when "last write wins" is the default, read it as what it is:

6. Residual risk — what you are still exposed to

You can shrink clock error; you cannot delete it. The honest engineering answer is to measure the uncertainty and then wait it out, which is what a globally distributed database does when it wants ordered transactions without a central sequencer:

"If we assume a quartz drift of 200 ppm, and it has been 30 seconds since the last clock sync, this implies a clock uncertainty of 6 ms due to quartz drift (on top of any uncertainty from network latency, GPS receiver and atomic clocks)."

Kleppmann, on TrueTime's uncertainty interval

That is the shape of the only correct answer: a timestamp is not a point, it is an interval, and you may only claim one event preceded another when their intervals do not overlap. Buying a small interval takes atomic clocks and GPS receivers in every datacentre. You do not have those. So the three things to carry out of this lesson:

• Export clock skew as a metric on every node, and alert on it. A node in NTP's panic state looks perfectly healthy from every other angle.
• Grep for wall-clock subtraction. Any now() - start using a time-of-day clock is a latent negative duration. Timeouts, leases, backoff and rate limiters are all durations.
• Treat every last-write-wins setting as a data-retention decision, and make someone own it. If losing a write is unacceptable, the fix is versioning, not tighter NTP.

7. Check yourself

8. Back to your world

Two searches, today. First, every subtraction of two wall-clock readings in your code — each one is a timeout, lease or latency figure that can go negative. Second, every place a row carries an updated_at that something later compares against another row's updated_at to decide which version to keep. The first is a bug waiting for an NTP step; the second is already losing writes, and it will never appear in your error rate.