How Complex Systems Fail:
An SRE Perspective.

Every production system ships with latent hazard: data loss, cascading failure, credential leak, silent corruption. You don't add the hazard by deploying; it was there the moment you chose distributed state.

SRE work is not the elimination of hazard, it is the continuous containment of it.

The reason your platform doesn't melt down daily is that it is layered with defenses: retries, circuit breakers, quotas, health checks, canaries, multi-AZ, multi-region, human review, change freezes, alert routing. Most of these defenses are invisible when they work. You only notice them when one is missing.

Nobody gets paged because of one bad commit. They get paged because a bad commit merged on a Friday during a deploy freeze exception, shipped through a canary with a broken metric, into a region whose autoscaler was already saturated, while the on-call was at dinner with their phone on silent.

Single points of failure exist (an expired TLS cert, a corrupted config), but the catastrophe still required the surrounding process defenses (monitoring, runbooks, ownership) to fail too. The single point is necessary; it is rarely sufficient.

Your system right now contains dozens of bugs, misconfigurations, expired-but-cached credentials, drift between IaC and reality, and dependencies pinned to versions that have since been yanked. You don't know which ones. You will learn about some of them during the next incident.

The production system you imagine in your head, where every pod is healthy, every queue is drained, every replica is in sync, does not exist and never has. Real systems always have something broken.

The art is running usefully despite constant partial failure.

The same system that served a billion requests today can serve zero tomorrow. Nothing about yesterday's uptime grants you tomorrow's. This is why leaders who say "we haven't had an outage in 90 days, we must be doing something right" terrify experienced SREs.

There is no root cause. There is a tree of contributing factors, and the node you pick to call "root" says more about your organizational incentives than about the system. Good post-mortems resist that choice for as long as possible.

Take the 2017 S3 us-east-1 outage. The "root cause" was an engineer typing the wrong argument. But equally true: no confirmation prompt, blast radius unbounded, subsystem hadn't been restarted in years, dependent services had no fallback. Pick all of them and you change how the system is built.

After the incident, the signal looks obvious. "CPU was climbing for 20 minutes, why didn't anyone see it?" Because at the time, CPU was climbing on 400 other dashboards too, and 399 of them resolved themselves. Hindsight collapses a fan of possibilities into a single narrative arc. Evaluate decisions by what was knowable at the time, not what the timeline reveals later.

Cook's framing doesn't map cleanly onto a modern SRE org chart. You'll find pure producers (product engineers), pure defenders (platform teams), and hybrids. The dual role isn't a property of every individual, it's a property of the system as a whole.

The faster the org ships, the more defense it owes. Organizations that pretend this tradeoff doesn't exist pay for it in burnout or outages, usually both.

Every deploy, every rollback, every kubectl delete pod, every "let me try restarting it" is a bet against a system you cannot fully observe. Most bets pay off. The ones that don't become incidents. The goal is not to stop gambling, it is to make the gambles smaller and more reversible.

Dashboards, runbooks, and alerts give you signals, not answers. At 3 AM, when 429s are spiking in eu-west-1 only for authenticated traffic, no document tells you whether to roll back, drain the region, or wait it out. The on-call engineer is the one who decides, and that decision is what actually changes the system. All the prep work is judged by whether it sharpens that decision or muddies it.

When the incident is novel, tooling cannot save you. A human decides to failover, to drain a node, to call the vendor, to accept the data loss and restore from backup. Automation handles the known; humans handle the unknown.

This is why the right framing for AI in SRE is assistive, not autonomous. AI that hands an on-call engineer better context faster makes the human decisive. AI that tries to replace the human decision makes the human unaccountable.

The expert on your payments pipeline last year is not the expert this year, because the pipeline has changed and so has the expert. Expertise is perishable. Tribal knowledge is the most expensive form of knowledge: it leaves with the person. Documented, searchable, queryable system knowledge is the antidote.

Every deploy, every migration, every provider switch, every version bump opens a new failure surface. The stability you feel right before a major change is partly an illusion: you have simply not yet discovered the failure modes the change will introduce. Plan rollbacks accordingly.

If your post-mortem concludes "engineer pushed bad YAML," your remediation is "more review." If it concludes "our validation pipeline didn't catch an invalid resource spec," your remediation is "better validation." Same incident, very different defenses. Where you locate the cause determines what defenses you build.

You cannot buy safety by buying safe components. A 99.99% service composed of 99.99% components is not 99.99% reliable; the composition matters. Reliability is an emergent property of the whole, which is why "we use AWS, so we're fine" is not a reliability strategy.

Buying safer components is necessary, just not sufficient. Good components don't compose into a safe system on their own. You still have to design for how they fail together.

The reason your system is up right now is that, at this moment, dozens of people are quietly choosing not to do risky things, catching near-misses in review, noticing a weird metric, and fixing a fragile deploy. This invisible labor is the bulk of reliability work, and it never shows up in any dashboard.

The loud hero breaks something on Friday and gets a Slack post. The quiet hero prevented the break on Tuesday by rewriting a fragile script and gets nothing. Most orgs reward the first and depend on the second.

Teams that never fail never learn. Game days, chaos engineering, and controlled failure injection generate the experience that pure uptime denies you. A team that has rehearsed its failover will execute it. A team that has only read the runbook will not.

You don't need real failures, you need exposure to failure. A team that hasn't had a real incident in two years can mistake that for resilience. The first real one will be larger and less informed precisely because nobody has practiced.