You add one field to a response. A week later, a mobile app three versions old starts crashing on launch.
Nothing about the new field was wrong. The problem is that the data had to survive a journey — out of memory, into bytes, across a network, into a different program written at a different time in a different language — and the rules for that journey were never agreed.
Every system that stores anything or sends anything anywhere does this constantly, and the failures show up long after the change, in software you no longer control.
In memory, and on the wire
Inside a running program, data is a structure the language understands: objects with pointers, hash maps, references. That representation is useless anywhere else. It is tied to one process's memory and often to one language.
To store it or send it, you encode (or serialise) it into a sequence of bytes. To use it again, you decode it back.
Diagram, as text
- Object in memory leads to Encode
- Encode leads to Bytes on disk or wire
- Bytes on disk or wire leads to Decode
- Decode leads to Object in another process
The whole subject is about that middle box: what the bytes mean, and who has to agree on it.
Text formats: readable, and larger
JSON is the default for web APIs. It is human-readable, every language can parse it, and you can debug it by eye.
The costs are real but usually acceptable:
- Size. Field names repeat in full on every record. A million records with a
created_atfield carry that string a million times. - Ambiguity about numbers. JSON has one number type, and large integers lose precision when parsed as floating point in some languages.
- No schema by default. Nothing states which fields must exist or what type they are, so mistakes are discovered by the reader, at runtime.
CSV is smaller and simpler and has no types at all — everything is text, and the meaning of a column lives entirely in a header row or a person's memory.
Binary formats: smaller, and schema-driven
Binary encodings replace field names with numeric tags and store values in compact form. A record that is 200 bytes of JSON is often 60 in binary.
The important difference is not size. It is that binary formats are defined by a schema — a separate declaration of the fields, their types and their tag numbers, agreed by both sides and checked by tooling:
message User {
required int64 id = 1;
optional string email = 2;
}The number after each field is the tag, and it is what actually travels. The name email is compiled away. That single design choice is what makes safe evolution possible, and it is the reason this lesson exists.
The size difference is not academic. The same small record, encoded four ways:
One record, four encodings
The data as a table
| Size | Bytes on the wire |
|---|---|
| JSON | 220 B |
| JSON (gzip) | 140 B |
| MessagePack | 150 B |
| Schema-driven binary | 60 B |
Multiply that by a billion messages a day and the choice stops being about taste. But notice what the smallest format costs you: the bytes are meaningless without the schema, which is why the rest of this topic is about keeping that schema compatible.
The four kinds of compatibility
Once two programs exchange data, they will be upgraded at different times. Four properties describe what survives.
- Backward compatible — new code can read old data. Needed whenever you deploy: the database is full of records written by yesterday's version.
- Forward compatible — old code can read new data. Needed whenever a rollout is gradual: half your servers are new and half are old, and they are talking to each other.
- Full compatibility — both of the above.
- Breaking — neither. Requires a coordinated change, which in practice means downtime or a long dual-write migration.
Forward compatibility is the one people forget, and it is the one that causes the mobile-app crash in the opening. You cannot upgrade a phone in someone's pocket. Old readers will see new data for as long as those installs exist.
The rules that keep you compatible
They are short, and they follow directly from tags travelling instead of names.
Adding a field is safe if it is optional. New readers see it; old readers do not recognise the tag and skip it. If you make it required, every old writer produces data new readers reject.
Removing a field is safe only if it was optional. Removing a required field breaks every reader that still expects it.
Never reuse a tag number. If field 7 was phone and you delete it, tag 7 must be retired forever. Reusing it for country means old records decode their phone number into the country field — no error, just silently wrong data, which is far worse than a crash.
Never change a field's type. Changing int32 to string means old bytes are reinterpreted as something else. Add a new field instead and migrate.
Where the schema lives
With JSON, the schema is usually in a document, or in people's heads, and enforced by neither. With binary formats it is a file that both sides compile against.
For data flowing through queues and streams, a schema registry is the common answer: writers register the schema, each message carries a small identifier, and readers fetch the matching schema to decode. It moves compatibility from a convention into a check that can fail in a build rather than in production.
Check your understanding
Common mistakes
Using language-native serialisation across a boundary. It couples the data to one language and one class definition, and decoding untrusted input is a security hole.
Treating "add a field" as always safe. It is safe if it is optional and readers ignore unknowns. Adding a required field breaks every existing writer.
Only testing the new version against the new version. The interesting cases are new-reads-old and old-reads-new, and neither appears in a test suite that upgrades everything at once.
Assuming every client upgrades. Mobile installs, cached scripts and third-party integrations persist for years. Old readers are permanent, not temporary.
Storing numbers in JSON without thinking about precision. Large identifiers exceed what a double can represent exactly, and the value changes without any error being raised.
Practice
A user record currently has `name` as a single string. You need to split it into `first_name` and `last_name`. Design a migration that never breaks a reader or a writer, and say how long each stage lasts.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
You cannot change a field's meaning in place. You can add, dual-write, and eventually retire.
Show a strong answer
Change it in four stages, and never mutate the original field's meaning.
- Add the new fields as optional. Do not touch
name. New writers populate all three;nameremains the source of truth. This is backward and forward compatible immediately, because every reader either sees the field it expects or skips one it does not. - Backfill. Run a job that derives
first_nameandlast_namefor existing records and writes them alongside the untouchedname. Splitting names is imperfect, so log the ambiguous cases rather than guessing silently — this stage is where the data quality problem is discovered, not the schema problem. - Move readers over, one at a time. Each reader switches to the new fields, falling back to
namewhen they are absent. Once every reader has been deployed and the fallback stops being hit — measure this, do not assume it — the new fields are authoritative. - Retire
name. Stop writing it, wait long enough that no client in the wild still reads it, then delete it. Its tag number is retired permanently.
How long each stage lasts is decided by the slowest reader you do not control. For internal services, days. For a mobile app, the honest answer is until usage of old versions falls below what you are willing to break — often a year or more, which is why stage 1 must be designed to be survivable indefinitely.
Your team must choose between JSON and a schema-driven binary format for a new internal service handling 50,000 messages a second at about 800 bytes each. Argue both sides, then decide.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Work out the bandwidth first, then ask what a wrong field costs and when you would find out.
Show a strong answer
The numbers first. 50,000 × 800 B = 40 MB/s, about 3.5 TB a day. A binary encoding typically cuts that by half to two thirds, so the saving is roughly 20 MB/s — real, but not by itself decisive at this scale.
For JSON: every language reads it, you can inspect a message with no tooling, debugging is possible from a log line, and there is nothing to compile or distribute. For a service two teams own and iterate on weekly, that friction saving is genuine.
For binary with a schema: the schema is a checked contract. A field removed or retyped fails at build time rather than at 3 a.m. in a consumer nobody remembered. Encoding and decoding are faster, which at 50,000/s shows up as CPU. And the compatibility rules are enforced rather than remembered.
The decision hinges on how many consumers there are and how independently they deploy. One producer and one consumer, same team: take JSON, and revisit if the bill or the CPU becomes visible. Several consumers deploying on their own schedules: take the schema, because the value is not the bytes saved — it is that incompatible changes become impossible to ship by accident.
For this case, at 50,000/s, an internal service almost certainly has several consumers and a long life. I would take the schema-driven format, and keep a debugging path that can print a message as text.
Key takeaways
- Encoding is the boundary between one process's memory and everything else. Data crossing it outlives the code that wrote it.
- Text formats are readable and self-describing but larger and unchecked; binary formats are compact and defined by a schema that tooling can enforce.
- Tags travel, names do not. That is what makes evolution possible, and why a reused tag number silently corrupts data.
- Backward compatibility covers deploys; forward compatibility covers rollouts and clients you cannot upgrade. The second is the one that gets forgotten.
- Add optional fields, never remove required ones, never reuse a tag, never change a type, and preserve unknown fields on read-modify-write.
Further reading
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), chapter 4 — the definitive treatment of encoding and schema evolution.
- RFC 8259, The JavaScript Object Notation (JSON) Data Interchange Format — including the number-precision caveats.
- Andrew Birrell and Bruce Nelson, "Implementing Remote Procedure Calls" (ACM TOCS, 1984) — where marshalling across a boundary was first set out.
- Rich Hickey, "Spec-ulation" (Clojure/conj, 2016) — a clear argument that growth is safe and breakage is not.
- Ken Thompson and Rob Pike, "UTF-8 history" (Usenix, 1993) — how text itself came to have a stable encoding.