Data evolution

Evolve data without lockstep

A durable data format lets old and new code coexist while a system changes.

Inspired by Designing Data-Intensive Applications by Martin Kleppmann. This is an original explanation, not a reproduction of the text.

The idea

Applications rarely upgrade all at once. During a rolling deployment, old and new servers run together. A mobile app may remain outdated for months. Stored records can outlive every program version that created them. That means a schema is not merely today’s object shape; it is a compatibility agreement across time.

Forward compatibility asks whether older code can handle data written by newer code. Backward compatibility asks whether newer code can read older data. Safe evolution usually favors changes readers can tolerate: adding optional fields, supplying sensible defaults, and preserving the meaning of existing fields. Renaming, retyping, or repurposing a field demands a deliberate migration.

Make it concrete

An order event originally contains an order ID and total. A new producer adds an optional delivery window. Updated consumers use it; old consumers ignore what they do not understand. New consumers also accept old events with no window and apply a default. If the team instead changes total from dollars to cents under the same field name, both versions can parse the number yet disagree about its meaning—a compatibility failure that syntax checks miss.

Change in two phases

First make readers tolerant of both representations and deploy them broadly. Then change writers. Remove the old representation only after stored data, downstream consumers, and long-lived clients no longer depend on it. Test mixed-version combinations, not just the newest producer with the newest consumer.

Keep this: Treat schemas as contracts between versions. Preserve meaning, make additions tolerable, and separate “read the new form” from “write only the new form.”

Try it

Imagine adding one field to an event your team owns. What will an old reader do with it? What will a new reader do when it is absent? Write those two behaviors before changing the producer.