Stream processing

A small, self-contained, immutable record that something happened at some point in time.

Events are generated once by a producer and processed by many consumers. Thinking of data as a sequence of events, rather than as mutable state, is the foundation for everything else in the chapter.

A server that producers send events to and consumers receive them from, decoupling the two.

Brokers tolerate clients coming and going and absorb bursts. Two key design questions: what happens when producers are faster than consumers, and what happens if a node crashes.

Blocking the producer when the consumer cannot keep up, instead of dropping messages or buffering without bound.

The three options for an overloaded consumer are drop, buffer, or block. Buffering only postpones the problem and raises latency; dropping loses data; backpressure slows the whole pipeline to the speed of its slowest part.

Load balancing delivers each message to one of several consumers; fan-out delivers every message to all of them.

Load balancing parallelizes expensive processing; fan-out lets independent consumers each see the whole stream. Consumer groups in log-based brokers combine both.

A consumer acknowledges each message after processing; unacknowledged messages are redelivered if the consumer dies.

Redelivery prevents loss, but combined with load balancing it reorders messages: a redelivered message can reach another consumer after later ones were already processed.

A broker that stores messages in an append-only, partitioned log on disk; consumers read sequentially and messages are not deleted on consumption.

It gives durable, replayable, totally ordered (per partition) streams with high throughput. Parallelism is at the granularity of partitions, and a slow message holds up its whole partition.

Each consumer group records how far it has read in each partition as a single offset number.

Offsets make acknowledgment cheap and replay trivial: rewind the offset to reprocess. A consumer that lags further behind than the retention window loses the messages that were deleted.

Application code writing the same change to several systems (database, cache, search index) itself.

Two concurrent clients can have their writes arrive in different orders at each system, leaving them permanently inconsistent, and one write can fail while the other succeeds.

Observing every write to a database and exposing it as a stream that other systems can apply in the same order.

It makes the database the leader and derived systems followers of its replication log, so they converge on the same state. Initial state comes from a snapshot or a compacted log.

Periodically discarding older log entries for a key, keeping only the most recent value (and dropping keys with a tombstone).

The log then stays proportional to the size of the database rather than to its write history, yet a new consumer can still rebuild the full current state by reading it from offset 0.

Storing every change to application state as an immutable event in an append-only log, and deriving current state by replaying it.

Events capture intent at the application level, history is never lost, and new read-optimized views can be derived from the same events. Snapshots keep replay fast.

A command is a request that may still be rejected; once validated it becomes an event, an immutable fact.

Validation must happen synchronously before the event is written. Consumers of the event cannot reject it, only compensate with further events.

Event time is when the event occurred (by the device's clock); processing time is when the stream processor sees it.

Queueing, network faults, and restarts delay events, sometimes by minutes. Windowing by processing time produces artifacts such as spikes after a backlog, so rates should be computed by event time.

A straggler is an event that arrives after its window was believed complete; a watermark is the processor's declaration that no earlier events are expected.

You can never be sure all events for a window have arrived. Either ignore stragglers (and monitor how many you drop) or publish a correction later.

Tumbling (fixed, non-overlapping), hopping (fixed, overlapping), sliding (all events within an interval of each other), session (bounded by inactivity gaps).

Aggregations over an unbounded stream need a bounded scope. The window type determines which events are counted together and whether one event can count in several windows.

Joining events from two streams that share a key and occur within some window of each other, e.g. searches and clicks.

The processor must keep state, an index of recent events from both sides, and the window decides when it stops waiting and emits a non-match.

Augmenting each event with data from a table, using a local copy of the table kept current by a changelog stream.

Avoids a remote database query per event. The join is time-dependent: an event joined before or after a profile update gets different data.

Two changelog streams joined to maintain a materialized view of the join, such as a per-user timeline cache of tweets from followed accounts.

Instead of computing the join at read time, every tweet, delete, follow and unfollow incrementally updates the cached result.

Breaking a stream into small blocks processed like mini batch jobs, or periodically checkpointing operator state and input offsets.

On failure the processor restarts from the last checkpoint and discards partial output, but any side effects already sent to external systems would be repeated.

An idempotent operation has the same effect whether it is applied once or several times; with it, at-least-once processing yields effectively exactly-once results.

Storing the input offset alongside the output, or committing output and offset atomically, makes reprocessing after a crash harmless.