Reliable, scalable, maintainable applications
Faults are routine; failures are optional
reliabilityfault-vs-failurehardware-faultssoftware-faultshuman-errorschaos-engineeringA fleet of 1,200 disks
day 0- no incidents yet
Each disk fails independently with probability 1/MTTF per day and is replaced after the repair time. Try: play with 1 copy and watch every hardware fault become data loss; switch to 3 copies and the same faults are masked. Then press "software bug": a correlated fault takes down all replicas at once.
Toy network: click a left dot to tweet, a right dot to load a home timeline
Amber dots are celebrities (≥6 followers); in hybrid mode their tweets are not fanned out (dashed) and are merged in at read time. Try: tweet as the biggest account in each mode, then raise the skew and run traffic.
Toy: cumulative operations
Operations per second: 100M users × 200 follows, 4.6k tweets/s, 300k timeline reads/s
Fan-out on read costs 300k × 200 = 60M lookups/s. Fan-out on write moves the work to tweet time, which is cheaper on average because reads outnumber tweets, but one celebrity tweet becomes tens of millions of inserts. Try: set skew to 0 (no celebrities), then to 0.75 and watch the worst-case delivery time.
Response-time distribution, last 2,000 requests (log scale)
Each bar is one request (newest on the right)
Even with identical requests, response time varies: context switches, packet loss, GC pauses, cache misses. Try: raise the stalls to 3% and watch the mean jump while the median barely moves; the p99 and p999 are what your heaviest users (often your best customers) feel.
Tail latency amplification
tail-latency-amplificationOne user request fans out; it is as slow as its slowest call
Red = slow call (>200 ms), purple = rescued by a hedged backup request. Try: keep 1% slow calls and drag fan-out to 100; most user requests become slow even though 99% of backend calls are fast.
P(user request is slow) = 1 − (1 − p)^N
A FIFO request queue in front of the server
t = 0.0 sLast 100 responses: waiting (amber) + processing (blue / red)
A few slow requests hold the only worker, and every fast request queued behind them pays the wait. Try: inject a 450 ms request with 1 worker, then with 4. Then switch to a closed-loop load generator: it stops sending while it waits, hiding the queue that real users would build up.
One day of load vs capacity: elastic: autoscale
One node serves 1k req/s at $0.20/h; a machine k times bigger costs k^1.5 times as much and tops out at 24×. Try: turn on the failure, which takes the single big machine down for 20 minutes but only one node of a fleet; then shorten the autoscaler's boot time to catch the viral spike.
Click a module to change it: 29 other modules may break
Arrows point from caller to callee. A change ripples to everything that depends on the changed module, directly or transitively. Green rings are interfaces: they hide implementation details, so a change behind one stops there. Try: add modules in the ball of mud and watch the average blast radius grow; then click an interface (a public API change) in the modular design.
- takeawayFaults happen all the time at scale. Design so they don't become failures: redundancy for random hardware faults, and for correlated software and human faults use testing, isolation, gradual rollouts and fast rollback.
- takeawayDescribe load with the parameter that dominates cost (for Twitter, fan-out), and describe performance with percentiles of client-side response time. Tails grow with fan-out and with queueing.
- takeawayScaling is a set of trade-offs between big machines, many machines and elasticity. Maintainability comes from removing accidental complexity with abstractions that keep a change from spreading.
Key concepts · 20
book chapter 1- Reliability
The system keeps working correctly, at the expected performance, even when things go wrong.
Things going wrong is the normal case at scale. Reliability is designed in, by expecting faults and keeping them from turning into user-visible failures.
↑ see it in the visualization - Fault vs failure
A fault is one component deviating from its spec; a failure is the system as a whole no longer serving the user.
You can't drive faults to zero, so fault-tolerant design aims to stop faults from becoming failures. Redundancy masks a dead disk; nothing masks a fault that hits every replica.
↑ see it in the visualization - Hardware faults
Disks crash, RAM goes bad, power and network fail. Mostly random and independent.
A disk's MTTF is 10 to 50 years, so a cluster of 10,000 disks loses about one a day. Redundancy (RAID, replicas, multi-machine setups) handles them, and software fault tolerance lets you lose whole machines.
RAIDAWS instance retirement↑ see it in the visualization - Software errors (systematic faults)
Bugs and runaway processes that are correlated across nodes because every node runs the same code.
They cause many more simultaneous failures than hardware does (for example the 2012 leap-second bug that hung Linux servers). Replication doesn't help; testing, isolation, monitoring and crash-restart design do.
Leap second 2012cascading failures↑ see it in the visualization - Human errors
Operator mistakes, such as bad configuration changes, are a leading cause of outages.
The fixes are about design: interfaces that make the right thing easy, sandboxes, fast rollback, gradual rollouts to a small canary group, and detailed telemetry.
↑ see it in the visualization - Deliberate fault injection
Killing processes or machines on purpose to make sure fault-tolerance machinery really works.
Error-handling code that never runs is probably broken. Exercising it continuously builds confidence that real faults will be handled.
Netflix Chaos Monkey↑ see it in the visualization - Scalability
Whether there are reasonable ways to deal with growth in load, data volume or complexity.
It is not a yes/no label. The useful question is: if load grows in this particular way, what are our options for adding resources?
↑ see it in the visualization - Load parameters
The few numbers that describe load for your system: requests/s, read/write ratio, active users, cache hit rate.
Choosing the right one matters. For Twitter the tweet rate was not the bottleneck; fan-out (followers per user) was.
↑ see it in the visualization - Fan-out (home timelines)
Building home timelines either by merging followees' tweets at read time, or by pushing each tweet into every follower's cached timeline at write time.
Pushing on write makes reads cheap but turns a celebrity's tweet into millions of writes. Twitter ended up with a hybrid: fan out most users' tweets, and merge celebrities' tweets in at read time.
Twitter↑ see it in the visualization - Throughput
Records processed or requests served per second.
Batch systems care mostly about throughput; online systems care mostly about response time. The two are linked through queueing.
↑ see it in the visualization - Response time vs latency
Response time is what the client sees (network + queueing + service). Latency is the time a request spends waiting to be handled.
Measure response time at the client. Time measured only inside the server misses the queueing delay.
↑ see it in the visualization - Percentiles (p50, p95, p99, p999)
The response time that a given share of requests is faster than. The median (p50) is the typical case.
The mean doesn't tell you how many users waited. High percentiles (tail latencies) hit the users with the most data, often the most valuable customers.
↑ see it in the visualization - SLOs and SLAs
Service-level objectives and agreements, often stated in percentiles: e.g. p50 under 200 ms and p99 under 1 s.
Percentiles make the contract measurable: customers can ask for a refund if it is not met.
↑ see it in the visualization - Tail latency amplification
When one user request makes N backend calls in parallel, it is as slow as the slowest call.
P(slow) = 1 − (1 − p)^N, so a 1% slow backend makes most user requests slow once N reaches the hundreds.
Google's "Tail at Scale"hedged requests↑ see it in the visualization - Head-of-line blocking
A few slow requests hold the server's limited workers, and the fast requests queued behind them wait.
This is why client-side response time can be far worse than server-side processing time. Load tests must send requests independently of responses (open loop), otherwise they hide the queue.
↑ see it in the visualization - Scaling up vs scaling out
Vertical scaling moves to a more powerful machine; horizontal scaling spreads load over many small machines (shared-nothing).
Big machines get expensive fast and are a single point of failure; distributing stateless services is easy, but distributing stateful data adds a lot of complexity. Real systems mix the two.
↑ see it in the visualization - Elasticity
Adding computing resources automatically when load increases.
Useful when load is unpredictable, but autoscalers react with a delay. Manually scaled systems are simpler and have fewer operational surprises.
AWS Auto ScalingKubernetes HPA↑ see it in the visualization - Operability
Making it easy for operations teams to keep the system running smoothly.
Good monitoring, automation, documentation, predictable behaviour and sensible defaults let people focus on the high-value work.
↑ see it in the visualization - Simplicity and accidental complexity
Accidental complexity comes from the implementation, not from the problem the users have.
Abstraction is the main tool for removing it: a clean interface hides implementation details, so a change behind it doesn't ripple through the codebase.
↑ see it in the visualization - Evolvability
Making it easy to adapt the system to new requirements (also called extensibility or plasticity).
Requirements always change. How easily you can change a data system depends a lot on how simple its abstractions are.
↑ see it in the visualization