Lesson 13 · Data on one machine · Module 1

The Compatibility You Owe

A rolling deploy is not a moment, it is a window — and inside that window two versions of your code are both writing and both reading the same data. Every schema change you have ever shipped was a bet on what the other version would do with bytes it had never seen.

The win in this lesson: you will be able to look at a proposed schema change and say, with a reason, whether old readers survive it and whether old writers survive it — and name the one change that breaks nothing loudly and corrupts data quietly.

1. Two versions, always

Deploys are not atomic. You replace instances a few at a time, so for minutes — or, if the rollout is cautious, hours — version 3 and version 4 are both serving. Add queues, and the window is longer still: a message written by v4 can be consumed by a v3 worker that has not been restarted yet. Add a rollback, and the arrow reverses: v4 wrote data that v3 must now read forever.

So there are two separate obligations, and people mix them up constantly. Pin them down:

Two sentences to keep: backward is about old data, forward is about old code. And forward compatibility is never something you add later — it is a property the old binary already has or already lacks. By the time you need it, the code that must provide it has shipped.

2. What Protobuf actually puts on the wire

Open the bytes and the rules stop being folklore. A Protobuf message is not a document with labels:

"A protocol buffer message is a series of key-value pairs. The binary version of a message just uses the field's number as the key — the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file)."

Protocol Buffers documentation, Encoding · Message Structure

The field name is a comment. It exists in your .proto file and in your generated code, and it never travels. What travels is the number, packed together with a three-bit type hint:

tag = (field_number << 3) | wire_type       encoded as a varint

  wire type 0  VARINT   int32, int64, uint32, uint64, sint32, sint64, bool, enum
  wire type 1  I64      fixed64, sfixed64, double
  wire type 2  LEN      string, bytes, embedded messages, packed repeated fields
  wire type 5  I32      fixed32, sfixed32, float

"The 'tag' of a record is encoded as a varint formed from the field number and the wire type via the formula (field_number << 3) | wire_type. In other words, after decoding the varint representing a field, the low 3 bits tell us the wire type, and the rest of the integer tells us the field number."

Protocol Buffers, Encoding · Structure of a Record

And the sentence that the entire lesson hangs on:

"The wire type tells the parser how big the payload after it is. This allows old parsers to skip over new fields they don't understand. This type of scheme is sometimes called Tag-Length-Value, or TLV."

Protocol Buffers, Encoding · Message Structure

Better than skipping: proto3 keeps the bytes. "Proto3 messages preserve unknown fields and include them during parsing and in the serialized output, which matches proto2 behavior." So a v1 service that reads a message, changes one field and writes it back does not shred the fields v2 added. It carries them through blind.

3. The rules that follow

Now the rules are derivable rather than memorised. The docs state them plainly:

"Adding new fields is safe. If you add new fields, any messages serialized by code using your 'old' message format can still be parsed by your new generated code… Similarly, messages created by your new code can be parsed by your old code: old binaries simply ignore the new field when parsing."

Protocol Buffers documentation, Language Guide (proto3) · Updating A Message Type

That single paragraph covers both directions at once. New code reads old data because a missing field yields its default — "when a message is parsed, if the encoded message bytes do not contain a particular field, accessing that field in the parsed object returns the default value for that field". Old code reads new data because of the skip loop. Removal is the mirror image, with a condition attached:

"Removing fields is safe. The same field number must not used again in your updated message type. You may want to rename the field instead, perhaps adding the prefix 'OBSOLETE_', or make the field number reserved, so that future users of your .proto can't accidentally reuse the number."

Protocol Buffers, proto3 · Updating A Message Type

Why renaming is safe and why that is a trap

Renaming coupon to promo_code while keeping tag 3 changes not one byte. The wire does not care. But everything keyed by name does: your generated accessors, anyone parsing the proto3 JSON mapping, and any downstream consumer written against the old field name. "Wire-safe" is a claim about bytes, not a claim about your build.

Why enum values come with a catch

Adding an enum value is wire-safe — the bytes decode fine. What happens next is your problem, and the docs say so outright: "any wire-safe changes may be a breaking change to application code in a given language. For example, adding a value to a preexisting enum would be a compilation break for any code with an exhaustive switch on that enum." The encoding is compatible; the behaviour has to be made compatible by you, with a default branch that does something sane rather than something confident.

4. The change that breaks everything, quietly

Every failure so far is loud or harmless. Here is the one that is neither. The field number is not a name for the field — it is the field:

"This number cannot be changed once your message type is in use because it identifies the field in the message wire format. 'Changing' a field number is equivalent to deleting that field and creating a new field with the same type but a new number… Field numbers should never be reused. Never take a field number out of the reserved list for reuse with a new field definition."

Protocol Buffers, proto3 · Assigning Field Numbers

Suppose tag 7 was account_id, an int64, deleted last year. Someone adds retry_count, an int32, and picks 7 because 7 is free in the file. Both are wire type 0. Every archived message, every queued message, every replayed event still carries an account id under tag 7 — and the new parser will decode it, without complaint, as a retry count. It does not throw. It has no way to throw:

"Reusing a field number makes decoding wire-format messages ambiguous. The protobuf wire format is lean and doesn't provide a way to detect fields encoded using one definition and decoded using another. Encoding a field using one definition and then decoding that same field with a different definition can lead to: Developer time lost to debugging · A parse/merge error (best case scenario) · Leaked PII/SPII · Data corruption"

Protocol Buffers, proto3 · Consequences of Reusing Field Numbers

Read that list again in order. A parse error is the best case, because at least something stopped. The other outcomes are an account identifier arriving in a field that gets logged, or a number written back to a column that now holds the wrong thing forever. This is the answer to "what could a schema change possibly break" that most people cannot produce.

The docs draw the boundary explicitly, and it is a boundary about deployment, not about syntax: "Only make wire-unsafe changes if you know that all serializers and deserializers of the data are using the new schema." If you cannot prove that sentence true — and with queues, archives and rollbacks you usually cannot — the change is not available to you.

5. Avro takes the other road

Protobuf pays for compatibility with a tag on every field. Avro refuses to pay it at all:

"Binary encoded Avro data does not include type information or field names. The benefit is that the serialized data is small, but as a result a schema must always be used in order to read Avro data correctly… Therefore, files or systems that store Avro data should always include the writer's schema for that data."

Apache Avro 1.12.0 Specification, Data Serialization and Deserialization

There are no tags. The bytes are field values, concatenated, in schema order — undecodable without the exact schema that produced them. So Avro moves the whole problem up a level: the schema travels with the data (in the header of an object-container file, or via an identifier resolved against a registry), and compatibility becomes a negotiation between two schemas rather than a property of the bytes.

"We refer to the schema used to write the data as the writer's schema, and the schema that the application expects the reader's schema. Differences between these should be resolved as follows…"

Apache Avro 1.12.0 Specification, Schema Resolution

Three of those resolution rules are the whole of Avro's compatibility story for records:

Note what has changed and what has not. Fields are matched by name, not by number — "the ordering of fields may be different: fields are matched by name" — so Avro's contract is the name, and renaming is the dangerous act rather than renumbering. Avro offers an escape hatch Protobuf does not: "An implementation may optionally use aliases to map a writer's schema to the reader's… if data was written as a record with a field named 'x' and is read as a record with a field named 'y' with alias 'x', then the implementation would act as though 'x' were named 'y' when reading."

Neither is better. They are different bets about where you would rather be wrong. Protobuf assumes readers and writers are deployed independently and may never share a schema source, so it makes every message self-describing enough to skip. Avro assumes the schema is available at read time anyway, so it deletes the per-field overhead and buys back a real default mechanism — an Avro default is declared in the schema and can be any value, where a Protobuf default is fixed by the language: "for numeric types, the default value is zero".

6. Residual risk

You can follow every rule above and still lose data. Three ways, in order of how often they actually happen:

Round-tripping through anything. Preserved unknown fields are the mechanism that makes an old service safe to leave in the path. It is fragile:

"Some actions can cause unknown fields to be lost. For example, if you do one of the following, unknown fields are lost: Serialize a proto to JSON. Iterate over all of the fields in a message to populate a new message."

Protocol Buffers, proto3 · Retaining Unknown Fields

Both of those describe an ordinary gateway. A proxy that converts to JSON and back, or a mapper that copies field by field into a fresh message, silently drops every field added after it was written. The message still parses. The new field is simply gone, and nothing in any log says so.

Zero and absent are the same value. Under implicit presence a field set to its default is not serialised at all, so a receiver cannot distinguish "the client sent 0" from "the client did not mention it". Ship an int32 discount = 9 and you have no way to tell a deliberate zero from an old client. This is why explicit optional exists, and why you decide presence at field-creation time, not when the ambiguity bites.

The reserved list protects one file. It is a compiler check on the .proto you control. It does not reach a vendored copy, a fork, a hand-written parser, or a consumer who reconstructed the schema from example payloads. If the schema is shared, the protection is only as good as the sharing.

7. Check yourself

8. Back to your world

The next time a schema change appears in review, ask three questions in this order and refuse to move on until each has an answer.

Does an old reader survive this? Not "will it once we have deployed" — will the binary running right now, and the one you might roll back to, parse these bytes. Does old data survive this? Not just the database: the queue backlog, the event archive, the request in flight. And is any number or name being re-used? If a tag is reappearing in a Protobuf file, or a field is being renamed in an Avro schema without an alias, that is the whole review — everything else can wait.

Then make it structural rather than cultural. Put schema compatibility checking in CI, with the old schema fetched from the deployed version rather than from the branch, so the check compares what is running against what is proposed. A rule nobody can forget beats a rule everybody agrees with.