The future of data systems

The authoritative copy of the data, where each fact is written first and exactly once.

When two copies disagree, the system of record wins by definition. Being explicit about which dataset is authoritative and which are derived is the first step to integrating many tools sanely.

Caches, indexes, materialized views and models computed from the system of record by some transformation.

Derived data is redundant by design: if lost or wrong, you can recompute it. That property is what makes it safe to have many specialized stores for one dataset.

Every change is appended to one totally ordered log; each derived system consumes it in that order.

Ad-hoc dual writes from the application race and partially fail, leaving stores permanently inconsistent. With a log, all consumers apply the same writes in the same order, so they can only be behind, never contradictory — they converge.

Building a new version of a derived view by replaying the full history alongside the old one.

It turns schema migrations into a gradual, reversible process: run both views, shift a fraction of reads to the new one, and roll back at any point because the old view never stopped.

A batch layer periodically recomputes exact views from immutable events; a speed layer serves fast approximate updates in between.

It combined reliable batch reprocessing with low-latency streams, but you implement the logic twice in two frameworks and must merge their outputs, and the approximate path drifts between batch runs.

One stream processor handles both live events and the replay of history from the beginning of the log.

With a log that retains history and engines that support event time and exactly-once semantics, the same code can rebuild a view from scratch or keep it up to date — one implementation instead of two.

Treating the log as the database's core and its indexes, caches and views as separate derived consumers.

A database already does this internally: CREATE INDEX scans existing data then follows new writes. Unbundling exposes that mechanism so you can compose specialized storage and processing tools, loosely coupled through the log.

Expressing derived datasets as functions from input streams to output state, run by stream processors.

Instead of application code reading and writing a shared mutable database, each view is a deterministic function of the log. Adding a view means wiring a new consumer, not changing the writers.

Consumers that perform side effects (emails, notifications, calls to external services) in response to events.

Unlike pure views, side effects don't replay safely: rebuilding from offset 0 sends every email again. Side-effecting consumers need to start at the head or deduplicate on event ids.

Logging read requests into a stream and answering them with a stream–table join against the derived state.

It records exactly what each user saw and when, which helps tracking causality and auditing; it also lets reads be routed to the partition holding the data, like a distributed query.

Some guarantees can only be implemented correctly with the help of the application at both endpoints of the communication.

TCP removes duplicate packets and a transaction is atomic, but neither knows that a retried HTTP request is the same transfer. Lower layers' guarantees help performance but don't remove the need for an end-to-end check.

A unique request id minted by the client and carried all the way to the database, enforced with a uniqueness constraint.

Retrying is the only way to survive a lost response, and only an end-to-end id makes a non-idempotent operation like a money transfer safe to retry: the second attempt finds the id and does nothing.

Route all requests for the same value to one partition and decide them sequentially in log order.

A single-threaded processor per partition decides deterministically which request came first, with no locks or distributed transactions, and scales by adding partitions.

Timeliness: readers see an up-to-date state. Integrity: no corruption, no contradictory or lost data.

Violations of timeliness are temporary — wait and they heal. Violations of integrity are permanent until someone repairs them. Most applications need integrity strictly and can tolerate some lag.

Systems that preserve integrity using logs, deterministic derivation and end-to-end ids instead of synchronous cross-node coordination.

They keep working across datacenters and network faults, with better performance, while giving up only timeliness. Synchronous coordination can be reserved for the few places where it truly cannot be avoided.

Fixing a constraint violation after the fact: refund, reassign, or apologize to the customer.

Many businesses already tolerate brief violations (overbooked flights, oversold stock) because a cheap apology beats the cost of strict coordination. The question is how often it happens and how expensive each apology is.

Continuously checking the integrity of data — checksums, re-deriving views from the log, comparing replicas.

Disks rot, software has bugs, and checksums can vouch for a bug's output. Systems that are designed to be audited (immutable events, deterministic derivation) let you detect corruption early instead of trusting blindly.

Chaining each log entry to the hash of the previous one (or using Merkle trees) so tampering is detectable.

Rewriting an old event breaks every hash after it. The same idea underlies certificate transparency and blockchains — a way to make the log's integrity verifiable by others.

Automated decisions (loans, insurance, hiring, policing) driven by models trained on data.

Models can codify existing discrimination and create feedback loops, and the people affected rarely can appeal or even learn why. Engineers who build these systems share responsibility for their consequences.

Behavioural data collected for one feature can reveal far more, and lives on beyond its purpose.

Every stored record can be breached, subpoenaed or repurposed after an acquisition. Collect less, keep it shorter, and treat data as a hazardous material rather than an asset.