Chaos Engineering: A Practical Guide to Breaking Things Safely

Most teams learn how their system fails the same way: in production, at 2am, with customers watching. Chaos engineering flips that. Instead of waiting for the network to partition or a dependency to time out at the worst possible moment, you cause those failures yourself, on purpose, in a controlled window, and watch what happens. The failures still teach you the same lessons. They just teach you on a Tuesday afternoon instead of a Saturday night.

In 2026 this matters more than it used to, and there is a specific reason why.

The failure happens either way. Chaos engineering is choosing when.

What is chaos engineering?

Chaos engineering is the discipline of experimenting on a system by injecting controlled failures, in order to build confidence in its ability to withstand real ones. The canonical definition, along with the five advanced principles the discipline is built on, is maintained at Principles of Chaos Engineering.

The word chaos is misleading. There is nothing random about it. A good chaos experiment looks exactly like the scientific method: you establish what the system does when it is healthy, you form a hypothesis about what will happen when a specific thing breaks, you break that one thing in a controlled way, and you measure whether the system behaved the way you predicted. If it did, you have proof of resilience. If it did not, you have found a weakness before a customer did.

The point is never to break things for the sake of it. The point is to replace assumptions with evidence. Most teams believe their system is resilient. Chaos engineering is how you find out whether that belief is true, on your terms, instead of discovering it is false during a real incident. If you want the theory behind why that gap between belief and reality exists at all, our writeup on how complex systems fail covers it directly.

Why chaos engineering matters more in 2026

Two things have changed, and both push in the same direction.

First, systems got more complex and more distributed. A modern application is dozens of services, third-party APIs, managed databases, caches, and cloud primitives, all with their own failure modes. Nobody holds the whole thing in their head anymore, which means nobody can reason their way to confidence about how it fails. You have to test it, and you have to be able to see what happened when you do, which is why your observability platform is a prerequisite for chaos work rather than a nice-to-have.

Second, and this is the 2026 accelerant, AI-generated code is shipping faster than teams can reason about it. When code is written and merged faster than any human fully reviews it, the number of untested failure paths grows quickly. You end up with systems whose behavior under stress nobody has actually thought through, because nobody wrote all of it by hand. Chaos engineering is one of the few ways to find those unknown failure modes deliberately, rather than waiting for them to surface on their own.

Once you accept that failure is constant and continuous, testing for it stops being paranoia and becomes basic hygiene.

There is also a cost argument that lands with leadership. The expensive way to discover a weakness is during a real outage, with customers affected, revenue lost, and engineers pulled into a midnight scramble. The cheap way is a controlled experiment on a Tuesday afternoon that finds the same weakness with nobody watching and nothing at stake. Chaos engineering is, in that sense, a way to pull the cost of failure forward to a moment you control and pay far less for it. The failures are going to happen either way. The only choice is whether you schedule them or they schedule you.

Where it came from: Netflix Chaos Monkey and real examples

Chaos engineering as a named practice started at Netflix. As they moved to the cloud, they built a tool called Chaos Monkey that randomly terminated production instances during business hours, forcing engineers to build services that could survive a machine vanishing at any moment. Chaos Monkey grew into the Simian Army, a set of tools that injected larger failures, including simulating the loss of an entire availability zone. The philosophy was simple: if losing an instance is inevitable, make it a non-event by causing it constantly until the system stops caring.

Netflix was not alone for long. Amazon has run structured failure exercises, often called GameDays, where teams deliberately trigger failures to test both the systems and the humans responding to them. The GameDay insight is subtle and important: an outage is never just a technical event, it is also a test of whether the on-call engineers know what to do, whether the runbooks are current, and whether the alerts fire. Breaking the system on purpose rehearses the people as much as the machines. Slack has publicly described using chaos experiments to validate the resilience of its infrastructure before failures found its users first. Across all of these, the common thread is the same: the most reliable companies do not avoid failure, they rehearse it.

There is a cultural point buried in these examples too. Every one of these programs depends on a blameless posture. If breaking things on purpose gets someone blamed when it reveals a weakness, nobody will run the experiments, and the practice dies. The companies that do this well treat a failed hypothesis as a win, because it found a real gap safely, rather than as a mistake.

The takeaway for everyone else is that this is not exotic anymore. The tooling that Netflix had to build from scratch is now available off the shelf, which is where the tools come in.

How does chaos engineering actually work?

Every real chaos experiment follows the same four steps, and skipping any of them turns an experiment into just breaking things.

1. 1Define the steady state

   Before you break anything, agree on what healthy looks like, measured, not guessed. This is usually a business or system metric: successful requests per second, latency under a threshold, error rate near zero. The steady state is your baseline and your safety signal.

2. 2Form a hypothesis

   State, in advance, what you expect to happen when you inject a specific failure. For example: if we kill one of three service replicas, the other two absorb the traffic and the error rate stays flat. Writing the prediction down is what makes it an experiment instead of a guess.

3. 3Inject one failure, with a small blast radius

   Break exactly one thing, and contain how much of the system it can affect. Blast radius is the core safety concept in chaos engineering: you start with the smallest possible scope, ideally in staging or on a tiny slice of production, so that if your hypothesis is wrong, the damage is tiny. You expand the blast radius only as your confidence grows.

4. 4Measure and learn

   Compare what actually happened against your steady state and your hypothesis. If they match, you have earned real confidence in that failure mode. If they do not, you have found a weakness, and that is the whole point. Either way, you learned something true instead of assuming it.

The discipline is in doing these in order, one failure at a time, so that when something breaks unexpectedly you know exactly which injection caused it. Step one is where most programs quietly fail: if you cannot state your steady state as a number, you have nothing to measure the experiment against. The four golden signals are the usual starting point for picking that number, and the recovery metrics are what step four measures once the fault is in.

Controlling the blast radius

Blast radius is what separates a defensible experiment from a reckless one. You expand outward only as each ring earns your confidence.

The chaos engineering tools compared

The tooling has matured into a few clear options, each suited to a different kind of team. The failure modes they inject are the same ones that show up in real incidents, so it is worth reading a few worked Kubernetes investigations before you pick what to break first.

Chaos engineering tools at a glance

The Chaos Maturity Model

Most teams do not need a new tool. They need to know where they are and what good looks like next. This is a simple maturity model for a chaos engineering program, five levels from nothing to fully continuous.

• Level 0Ad hocMost teams start here

  No chaos practice. You learn how the system fails when it fails, in production. Most teams start here.

• Level 1Manual game daysStart here

  You run occasional, planned failure exercises, a scheduled afternoon where the team deliberately breaks something in staging and watches. Manual, infrequent, but a real start, and this is where most teams should begin.

• Level 2Scheduled experiments

  Chaos experiments run on a regular cadence against known failure modes, with defined steady states and hypotheses. The practice is now a habit, not an event.

• Level 3Automated in the pipeline

  Chaos experiments run automatically as part of CI/CD, so a change that weakens resilience is caught before it ships, the same way a failing test blocks a merge. Resilience becomes a gate, not an afterthought.

• Level 4Continuous and autonomousNetflix end state

  Experiments run continuously in production with automated safety controls, constantly probing for regressions in resilience. This is the Netflix end state, where failure is so routine the system genuinely does not care.

The value of the model is not to rush to Level 4. It is to know your level honestly and take the next step. A team at Level 0 running its first game day has improved more than a team at Level 3 buying another tool. If you want to score chaos drills alongside the other eleven reliability levers, our Availability Planner walks through exactly that.

Where AI SRE fits with chaos engineering

Chaos engineering and AI SRE solve two halves of the same problem, and it is worth being precise about the seam between them.

Chaos engineering deliberately creates a failure. But creating the failure is only half the exercise. The other half is understanding what actually happened: what cascaded, which dependency buckled, why the blast radius was larger than predicted, and what the real root cause of the unexpected behavior was. When a chaos experiment surprises you, and the useful ones do, you are suddenly running an investigation, just one you started on purpose.

That investigation is exactly where an AI SRE fits. A tool like Sherlocks AI does not run your chaos experiments and does not replace Gremlin or Chaos Mesh. It sits alongside them and does the part that comes after the injection: correlating the signals, tracing what the failure actually touched, and reaching a confirmed explanation for why the system behaved the way it did. Chaos engineering finds the weakness. The investigation explains it. They are complementary.

So the two fit together cleanly. Use a chaos tool to inject failure and find where your system is fragile. Then, when an experiment reveals something surprising, use the investigation layer to understand it fast, the same way you would for a real incident. For how that investigation works in practice, see our writeup on root cause analysis.

Chaos engineering FAQ

Related Reading