Encoding and evolution

Translating an in-memory data structure into a self-contained sequence of bytes, and back (decoding).

Pointers and objects only make sense inside one process. Anything written to disk or sent over a network has to be encoded, and the format you choose decides size, speed and how painfully your schema can change.

Built-in serializers such as Java Serializable, Python pickle or Ruby Marshal.

Convenient but they tie your data to one language, often allow decoding to instantiate arbitrary classes (a security hole), and treat versioning and efficiency as afterthoughts. Avoid them for anything that persists or crosses a team boundary.

Human-readable textual encodings that are widely supported across languages.

Good enough as a lingua franca between organizations, but verbose, ambiguous about numbers (integers above 2⁵³ break in JavaScript), lacking binary strings, and schema support is optional and complicated.

Binary encodings of the JSON data model, such as MessagePack, BSON or CBOR.

They shave some bytes and parsing cost, but because there is no schema every record still carries all its field names, so the savings are modest (81 → 66 bytes in the book's example).

Small numbers in a Thrift or Protocol Buffers schema that identify each field in the encoded bytes instead of its name.

Tags make records compact and make renaming a field free. In exchange the tag is forever: you may never change a field's tag or reuse a removed field's tag, because old data still carries it.

Schema-driven binary formats with code generation, from Facebook and Google respectively.

Thrift's CompactProtocol and Protobuf pack tag and type into one byte and write integers as variable-length varints, getting the example record down to 34 and 33 bytes.

Integers encoded 7 bits per byte with a continuation bit, often after zigzag mapping so small negatives stay small.

Most numbers are small, so most take one or two bytes instead of a fixed eight. A side effect: widening a field from 32 to 64 bits is wire-compatible, but old 32-bit readers silently truncate large values.

A Hadoop-born binary format whose encoding contains no tags and no types, only values concatenated in schema order.

It is the most compact (32 bytes), and because there are no tags, schemas can be generated from database tables and evolve freely. The cost: you cannot decode a single byte without knowing exactly which schema wrote it.

In Avro, the schema the data was encoded with and the (possibly different) schema the application expects when decoding.

Avro resolves the two side by side: fields are matched by name, so order doesn't matter; fields the reader doesn't know are skipped; fields the writer lacked get the reader's default. The writer's schema travels in the file header, a schema registry, or the connection handshake.

Newer code can read data that was written by older code.

Usually easy because the new code's author knows the old format. Rule: any field you add must be optional or have a default, otherwise new code fails on old records.

Older code can read data that was written by newer code.

Harder, because old code must tolerate additions it could not have anticipated: it has to ignore unknown fields, and you can only remove fields that are optional or have a default.

Changing a schema over time (adding, removing, renaming, retyping fields) while keeping old and new data mutually readable.

Each format has its own rules: tag-based formats tolerate renames but not tag changes; Avro tolerates reordering and tag-free generation but needs aliases for renames and defaults for any field it adds or drops.

Deploying a new version to a few nodes at a time, so old and new code run side by side for a while.

It enables zero-downtime deploys and quick rollbacks, but it means data flows in both directions between versions, so both backward and forward compatibility are required.

When old code reads a record, modifies it and writes it back, keeping the fields it didn't understand.

If old code decodes into a model object that drops unknown fields and then re-encodes, it silently destroys data written by newer code — without any error. Protobuf's generated classes keep unknown fields; many ORMs do not.

A database contains values written over years by many versions of the application, while only the latest code is deployed.

Rewriting (migrating) everything to the new schema is expensive, so most databases evolve lazily: old rows stay as they are and missing columns are filled in with defaults when read.

Making a request to a remote service look like calling a local function.

The abstraction is leaky: a network call can be lost, time out with an unknown outcome, take wildly variable time, and be retried — so without idempotence a retry may run the action twice. Treat a remote call as a different kind of thing, not as a function.

A design style built on HTTP's own features (URLs, methods, caching, content types), usually with JSON bodies.

Easy to experiment with and debug, supported everywhere, and good for public APIs. Compatibility is maintained by versioning the API, often in the URL or headers, because you can't force clients to upgrade.

An operation that has the same effect whether it is performed once or several times.

Retries are the only way to cope with lost responses, and they are only safe if the operation is idempotent, e.g. by attaching a unique request ID the server deduplicates on.

An intermediary that stores messages from producers and delivers them to consumers asynchronously.

It buffers when the recipient is slow or down, redelivers after crashes, decouples sender from recipient (sender doesn't need its address), and fans one message out to many consumers. The sender doesn't wait for a reply.

A programming model where actors with local state communicate only by asynchronous messages, transparently across nodes.

Message loss is assumed even locally, so going distributed changes less than with RPC, but rolling upgrades still need the message encoding to be forward and backward compatible.