Runbook Best Practices: Writing Runbooks Your On-Call Team Will Actually Use

Every team has a runbook problem. Some teams have too many runbooks that nobody reads. Some teams have too few. Some teams have runbooks that were technically correct in 2023 and have not been touched since. Almost every team has at least one runbook that everybody points to and nobody actually uses.

The advice most teams get about fixing this is generic. Write clear steps. Keep them updated. Store them where people can find them. All of that is true and none of it addresses why runbooks fail in the first place, which is much more specific and much more solvable. It is worth being precise about where runbooks sit inside reliability engineering before trying to fix them.

Runbooks fail because they are written for the wrong reader.

What is a runbook, and why do most of them fail?

A runbook is a documented procedure for handling a specific operational situation, most often an alert or an incident. In principle, it captures the steps an experienced engineer would take, so that a less experienced engineer, or the same engineer under stress, can execute them consistently. It is the tactical layer underneath a broader incident response process.

In practice, most runbooks are written and reviewed by people who will never have to execute them at 3am. The result is a document that reads well in a design review and falls apart the moment it meets a real incident.

The failure pattern is consistent across teams. The runbook says things like check the dashboard, or look at the logs, or verify the service is healthy. These instructions are meaningful to the person who wrote them. To an on-call engineer who was paged out of deep sleep, has already handled one incident tonight, and has never touched this specific service before, they are useless. The dashboard has fifty panels. Which one. The logs are twelve gigabytes per hour. Grep for what. Healthy how.

A runbook that requires the reader to already understand the system is not a runbook. It is expert-only documentation that assumes context the reader does not have. And the moment an engineer opens a runbook that fails them once, they stop opening runbooks. They go straight to kubectl and grep, which is slower and more error-prone, but at least it does not lie to them. Our analysis of 20 real investigations found the same thing from the other direction: the first alert rarely names the actual cause.

This is why most runbooks fail. Not because they are wrong. Because they were written for the wrong reader.

Who is a runbook actually for?

The unstated question every runbook writer should answer before writing a single step is: who is this for.

The wrong answers are common. Not the reviewer. Not the SRE lead who signs off on it. Not the auditor. Not the compliance team. Not a future engineer who might join the team next year.

Every design decision in the runbook, structure, length, tone, tooling, review cadence, ownership, follows from optimizing for that reader. A runbook that would score highly in a code review but fails that reader is a failed runbook. A runbook that looks sparse and undocumented but works for that reader is a successful runbook. The Google SRE workbook on incident response makes the same point about roles: under pressure, clarity beats completeness.

Get the reader right and everything else follows. Get the reader wrong and no amount of good intent will produce runbooks people use.

The seven rules of runbooks people actually use

Once you accept the 3am engineer as the primary reader, seven rules follow directly. None of them are new. All of them get violated constantly.

1. Optimize for the first minute, not the full incident

The first three actions in a runbook should be things the on-call engineer can execute without thinking. Verification steps, not remediation. The goal of the first minute is not to fix the problem. It is to confirm what is actually broken.

The good version does one thing. It has no ambiguity. It produces evidence the engineer can share. And it routes to the next step without requiring judgment. The Kubernetes guide to debugging a running pod is a good source for the exact commands worth pinning at the top.

2. Write in imperatives, not descriptions

Descriptions tell the reader what a good response looks like. Imperatives tell them what to do. Under stress, the difference matters enormously.

Every step in a runbook should read as a direct command. If a step needs the reader to infer intent, translate it to an imperative before you ship it. Where the command has flags worth knowing, link the upstream documentation rather than explaining it inline.

3. Every step needs a verification

After each action, the runbook should tell the reader exactly what output means success, what output means the next step, and what output means escalate. No ambiguity.

The second version tells the reader exactly what they should see, and what to do if they see something different. That is what an executable step looks like.

4. Decision trees, not paragraphs

Under pressure, humans process trees faster than they process prose. Every runbook that has more than three branches should be structured as an explicit decision tree.

If the log shows connection refused, go to section A. If it shows timeout, go to section B. If it shows neither, go to section C. Each of those sections is short and terminal, either resolving the incident or escalating.

Nobody reads paragraphs at 3am. Everybody can follow go to section A.

5. Every runbook has an explicit escalation path

The wrong version of an escalation instruction is escalate if you cannot fix it. That leaves the tired engineer to decide when to escalate, which is exactly the decision they are least equipped to make well.

Explicit exit paths do two things. They save trust, because the engineer knows the runbook has a defined edge, not just a hope. And they prevent long-tail incidents where a junior engineer keeps trying steps for two hours because nobody told them when to stop.

6. Test the runbook in a game day, not a review

A runbook that has never been executed under simulated pressure is not a runbook. It is a document. Nothing you learned about the runbook during the review process tells you whether it will actually work at 3am.

Real testing means running the runbook end-to-end during a scheduled game day. Someone plays the on-call engineer. Someone else plays the alert. The on-call engineer executes the runbook step by step, out loud, while a colleague verifies each step lands and notes where the runbook is ambiguous, missing context, or wrong. Every issue found in the game day gets fixed before the runbook goes back into rotation. If you have never run one, a step-by-step game day walkthrough is the fastest way to structure the first attempt.

Any team that has not game-day tested its runbooks in the last quarter is running on hope. The teams that make runbooks part of a monthly game-day rotation catch problems in a low-pressure environment instead of during a real 3am incident.

7. Every runbook has one owner, and if the owner leaves, the runbook is deleted

Runbooks decay. Systems change, alerts change, dependencies change, teams change. Without an owner, decay is silent and cumulative.

Assign one owner per runbook. When that person leaves the team, the runbook is either transferred to a new owner explicitly, or it is deleted. The failure mode most teams fall into is that runbooks accumulate as engineers rotate through, nobody feels responsible, and the runbook library slowly fills with documents whose factual accuracy nobody can vouch for. That library is worse than no library, because engineers still trust it enough to open it, and get burned when its instructions are stale.

Deleting a runbook feels like losing information. Keeping an unmaintained runbook is worse. Unmaintained runbooks actively erode trust in the whole library.

What most teams get wrong about runbook testing

The single most common mistake is confusing review with testing.

A runbook review checks whether the runbook makes sense to another engineer reading it in a normal state of mind. That is useful. It is not the same as testing.

Testing means executing the runbook, in the environment it will actually run in, with the person who will actually be on-call, under conditions that approximate real incident pressure. Game days are the standard way to do this, and they rest on the same premise as the principles of chaos engineering: you learn what a system does under failure by causing the failure deliberately. They are boring and they take an afternoon. Every team that runs them regularly discovers that their runbooks are missing steps, use commands the on-call engineer does not have permissions for, or route to services that were renamed six months ago.

The teams that skip game-day testing do not skip it because they do not believe in it. They skip it because it feels like an insurance cost that never pays off. Except during an incident, which is exactly when the untested runbook fails and the insurance would have mattered.

There is a simpler test worth doing between game days. Give the runbook to an engineer who has not read it before, ask them to execute it against a non-production replica of the failure mode, and time how long each step takes. If they get stuck, the runbook is wrong. If they finish in a reasonable time, the runbook probably works. This test costs half an hour and catches most bad runbooks before they hit real incidents.

How to structure your runbook library

Not every alert deserves its own runbook. The right starting number for most teams is two to three runbooks, covering the alerts that are either most frequent or most impactful.

Frequent runbooks reduce weekly toil, in the specific sense the Google SRE book uses the word. If your team gets paged for a database connection saturation alert three times a week, you need a runbook for it. Impactful runbooks reduce blast radius when the worst happens. If your team has an SLO on the checkout service and its failure would be catastrophic, you need a runbook for it even if it has never fired.

Grow the library from real postmortems, not proactively. When an incident happens and the responders spent 20 minutes figuring out what to check first, that is the signal to write a runbook. When an incident happens and the runbook worked but a step was missing, that is the signal to update it. A structured postmortem is what turns an incident into that signal. Runbooks written speculatively, without a real incident driving them, tend to describe failure modes that never happen while missing the ones that do.

Store the library where the on-call engineer already works. If your team lives in Slack, integrate runbooks with your alerting so the runbook link is one click from the alert. If your team lives in a specific documentation platform, keep them there. Do not force the on-call engineer to remember where a runbook lives at 3am. That memory is exactly what stress degrades first.

Whether any of this is working is a measurement question, not an opinion. Compare resolution time on alerts that have runbooks against alerts that do not, using the definitions in our guide to MTTR, MTTD, MTTA and MTTF. Teams that track this the way DORA treats failed-deployment recovery find out quickly whether the library earns its keep.

Where runbooks stop and AI SRE picks up

Even a great runbook only covers the failure modes you already anticipated. The runbook library is a memory of past incidents, curated into instructions. It cannot help with a failure that has never happened to your team before.

Novel failures are exactly the ones that make it into postmortems. A pod that gets OOMKilled while its node still has 96 percent free memory. A Postgres replica that throttles while its IOPS sits at 21 percent of provisioned. A cascading failure across three services that took an hour to correlate. These are the incidents where the responder opens the runbook library, finds nothing that matches, closes the wiki, and starts investigating from scratch.

That is the natural handoff to AI SRE. A tool like Sherlocks AI does not replace the runbook library. It handles the failures the runbook library cannot: the ones that require correlating signals across services, forming and ruling out hypotheses, and returning a confirmed root cause faster than a human can. When a novel failure resolves and becomes a postmortem, it also becomes a new runbook candidate, and the library grows in the right direction, driven by real incidents rather than speculation.

You can see how the investigation runs end to end on our examples page, where 20 real incidents are walked through with the full evidence trail, every signal pulled and every hypothesis ruled out.

Frequently asked questions

Further reading