We analyzed 20 real production incidents that we investigated end to end. In most of them, the first alert pointed at a symptom, not the cause. One pod was OOMKilled while its node had 96.88 percent free memory. A Postgres replica throttled hard on storage while IOPS sat at 21 percent of what was provisioned. A CloudWatch alarm fired on a batch job that no user ever noticed. The four patterns behind most of these incidents are consistent, and they explain why the dashboard looking fine is usually a statement about the wrong graph. Here is what actually breaks in production, with the real numbers.
20
Incidents investigated end to end
every one with a confirmed root cause
96.88%
Node memory free at the OOMKill
the graph that said nothing was wrong
862.8
MB/s against a 750 MB/s ceiling
while IOPS sat at 21% of provisioned
4
Patterns behind most incidents
container-local, alarm-only, workload, drift
A finding from the datasetA number that pointed the responder at the wrong layer
Most writing on production incidents is either abstract methodology or vendor case studies. This piece is different. It is a look at 20 real investigations, with real service names, real exit codes, and real metrics, and what the patterns across them say about how modern production actually fails.
The uncomfortable finding, and the one the rest of the article is built on, is this: the first alert in an incident rarely names the actual cause. The symptom is visible. The cause is one or two layers away. Every minute you spend on the layer that alerted is a minute you are not spending on the layer that broke.
How we did this
The sample is 20 incidents Sherlocks AI investigated end to end. Each has a full structured record: the service, the initial alert, every signal we pulled, the hypotheses we formed and ruled out, and the confirmed root cause with the specific metric or configuration that broke.
The mix by layer: Kubernetes 10, database (Postgres) 2, cache (Redis and ElastiCache) 4, storage and volume 2, VM 3. The remainder cross layers or sit on ECS and Aurora. Sample size is small, which we say up front. But every incident has real numbers, which almost no public dataset can say. This is a primary source, not a survey.
Counts as reported in the analysis. Incidents that cross layers are counted under each layer they touch, and the remainder sit on ECS and Aurora.
What we counted: the layer that fired the initial alert, the layer where the root cause actually lived, and the specific dimension or configuration that saturated or changed. What we did not count: qualitative factors like team size, on-call posture, or organizational maturity, because our data does not support those claims.
The headline finding: the alert almost never names the cause
Across the 20 investigations, the first alert pointed directly at the cause in only a small fraction of cases. Most of the time it pointed at the symptom, and the cause lived one or two layers underneath.
The clearest example: a pod evicted with an OOMKilled status. The natural instinct is to check node memory pressure. In the incident we investigated, the node had 96.88 percent free memory at the moment of eviction. The kill was container-local, driven by a specific query pulling 22.36 million rows through a Celery worker configured with a 2.5GB memory limit. The alert said memory. The cause was a query.
Where the alert fired
Pod OOMKilled
Exit 137 on the container. The instinct is to go look at node memory pressure.
Node memory free: 96.88%
Where the cause lived
An unbounded query
A single request pulled 22.36 million rows into memory, breaching the container's own limit.
Container limit: 2.5GB
This is not one weird incident. It is the pattern. A storage throttle looks like a database problem. A pool exhaustion looks like a connection problem. A CloudWatch alarm looks like an incident. Under each one is a different layer doing the actual breaking.
What layer actually breaks
Splitting the 20 investigations by where the alert fired versus where the cause lived produces a picture that most operators will recognize but almost no one has data on.
The layer the alert fired on skewed heavily toward the application: pod status, service errors, HTTP 503s, connection timeouts. This is what monitors are set up to catch, because it is what maps to user-visible symptoms.
The layer the cause actually lived in was almost never the same as where the alert fired. In the database and cache incidents, the trigger was frequently a workload-side event, an HPA scaling replicas, a deploy cycling pods, a schema migration removing an index. The database and cache did not spontaneously fail. Something upstream changed, and the failure surfaced on infrastructure that was itself healthy.
The single most useful mental shift these investigations forced is this: treat the alert as a claim to check, not a cause to fix. Read past the layer that alerted, because in most of the cases we investigated, that layer was fine.
The four failure patterns that keep repeating
Four patterns account for most of the 20 incidents. Each has a name, a real anchor from the investigations, and a one-line lesson.
Container-local versus node-level
Failures where the container's own configured limit is what saturated, while the underlying node is fine. This is the OOMKilled-with-96.88-percent-free-memory case. It is also the gp3 EBS throttling case: the volume enforces IOPS and throughput as two separate ceilings, and in the incident we investigated, throughput hit 862.8 MB/s against a provisioned 750 MB/s ceiling while IOPS sat at 21 percent of provisioned 30,000. The volume was healthy on the graph everyone watches. It was throttling hard on the graph nobody was looking at.
The lesson: name the exact dimension that saturated, not the service. A resource can be at 100 percent on one dimension and 20 percent on three others, so the volume looking fine is usually a statement about the wrong graph.
The alerting layer is the only thing broken
Failures where the alarm fired but no user-facing latency or error rate moved. In the RDS WriteIOPS incident we investigated, a batch job caused a short-lived write spike that tripped an alarm. The investigation confirmed no customer impact at all. The alarm was the incident.
The lesson: check every alarm against the request path before you treat it as an incident. Vacuum, backups, snapshots, and batch jobs all produce alarms that look alarming and mean nothing to a user.
The workload-side event that surfaces on the infrastructure side
Failures where the trigger was a client-side event, but the symptom appeared on the shared infrastructure it touched. In the two ElastiCache incidents, CurrConnections spiked from around 280 to 1,788 in one case (from HPA-driven pod churn) and from around 370 to 1,988 in the other (from a WebSocket reconnect and reject loop). Cache health, CPU, memory, and eviction rate all stayed normal in both incidents. The cache was healthy. The workload changed.
The lesson: decide whether the client or the service moved. Connection storms on a cache are frequently caused by pod churn, deploys, or an HPA cycling replicas, and the cache is the place where a workload-side event first becomes visible.
Migration and config changes that hide for weeks
Failures where a change shipped weeks earlier finally met the traffic that exposed it. The classic case is a schema migration that drops an index, and a query that used to run in 50 milliseconds now takes 42 seconds under real load. The connection pool exhausts because concurrency in use equals arrival rate multiplied by hold time, and a statement going from 4 milliseconds to 900 milliseconds multiplies the number of held connections by more than 200 at the same traffic.
The lesson: when one statement gets dramatically slower, look at what changed in schema or indexes, not at the traffic. The cause is rarely fresh. It is usually something that has been sitting there quietly, waiting for the right traffic to expose it.
The metric everyone watches, and the one that actually broke
Aggregating across the 20 investigations, one specific pattern stood out. In almost every incident, the metric the team was watching was not the metric that broke.
| Incident type | The graph everyone watches | The graph that actually mattered |
|---|---|---|
| gp3 storage throttling | IOPS utilization | Provisioned throughput (MB/s) |
| ElastiCache connection spike | CPU and memory | CurrConnections vs client-side pod count |
| Connection pool exhaustion | Max connections limit | Mean query duration and pg_stat_activity |
| Pod OOMKilled | Node memory pressure | Container memory limit and query row count |
| CloudWatch WriteIOPS alarm | The alarm itself | User-facing latency and error rate |
This table is the piece to screenshot, because it is the single most transferable lesson across the whole dataset. In every one of these incident types, the natural monitoring instinct pointed at the wrong signal.
What this means for how you investigate
Three concrete shifts follow from these 20 investigations, and applying them cuts investigation time meaningfully.
Read past the layer that alerted.
Treat the alerting layer as the surface where the symptom became visible, not as the layer to fix.
Name the specific dimension that saturated.
Not the cache is slow. CurrConnections spiked from 280 to 1,788 while cache health stayed normal.
Check the alarm against the request path.
If user-facing latency and error rate did not move, the alarm is a workload event, not an incident.
First, read past the layer that alerted. The alert almost never names the cause. Treat the alerting layer as the surface where the symptom became visible, not as the layer to fix. The AWS troubleshooting guides on our site are organized around exactly this principle: name the dimension, not the service.
Second, name the specific dimension that saturated. Do not say the cache is slow. Say CurrConnections spiked from 280 to 1,788 while cache health stayed normal. Precision on the dimension is what separates a real diagnosis from a guess, and it is the single practice that most consistently shortened the incidents we investigated.
Third, check the alarm against the request path. Before assuming an alarm reflects a real incident, look at whether user-facing latency or error rate moved during the same window. If they did not, the alarm is a workload event, and treating it as an incident wastes the on-call.
Together, these three shifts turn incident triage from pattern-matching against alerts into a much smaller, higher-signal search: which layer alerted, which layer actually moved, and which specific dimension broke.
Where AI SRE fits in this picture
Every one of the 20 investigations followed the same shape. Detection was fast. Alerting was fast. The time went into the correlation, the naming of the dimension, and the ruling out of layers that were healthy but noisy. Investigation is where the hour lives, and investigation is what the traditional stack has never automated.
That is where AI SRE fits, and specifically it is what Sherlocks AI does. The tool does not replace observability, monitoring, or incident management. It sits alongside them and does the correlation step: pulling the signals together, forming and ruling out hypotheses, and naming the specific dimension that broke, in minutes rather than an hour. Because it is tool-agnostic, it works on top of whatever observability platform, cloud, and orchestrator the team already runs.
The two decisions are separate. Pick the monitoring and observability that fit your stack, and if the slow part of your incidents is the correlation between the alert and the confirmed cause, add the investigation layer on top. Every one of the 20 investigations in this piece was a Sherlocks AI investigation, and every one is browsable on the examples page with the full evidence trail. If you want to see how a single incident plays out end to end, that is where to go.
For the general methodology behind the process, see our root cause analysis guide and the metrics that measure it in the MTTR guide. For the layer-by-layer troubleshooting patterns each of these incidents demonstrates, see the Kubernetes guides hub. For choosing the monitoring platform underneath the AI layer, see our observability platform guide.
The bottom line
The difference between a four-hour incident and a thirty-minute one is rarely skill or tooling. It is whether the first person on the call reads past the alert, names the specific dimension that broke, and checks whether the alarm reflects real user impact. Twenty incidents is a small sample. But every one of them, without exception, taught the same lesson: the alert is the symptom, and the layer underneath is where the fix lives.
Real incident data teaches this in a way opinion pieces cannot. And it is the single most useful discipline any on-call engineer can build, whether or not they add an AI layer on top.
Frequently asked questions
Across 20 investigations, the most common patterns were: container-local resource limits (not node pressure), workload-side events surfacing on shared infrastructure, alarm-only incidents with no user impact, and slow-onset schema or config changes exposed by traffic.
Because the kill is container-local. In one incident we investigated, a Celery worker was OOMKilled by its 2.5GB container limit while the node had 96.88 percent free memory. The cause was a query pulling 22.36 million rows, not node pressure.
Because gp3 provisions IOPS and throughput as two separate ceilings. In one investigation, throughput peaked at 862.8 MB/s against a provisioned 750 MB/s, while IOPS sat at 21 percent of provisioned. The volume was healthy on the graph most teams watch and throttling on the one they were not.
Almost always a workload-side event. HPA-driven pod churn or a client-side reconnect loop can push CurrConnections from a few hundred to nearly 2,000 in one bucket, while cache CPU, memory, and eviction rate stay perfectly normal.
No. Alarms can fire on batch jobs, snapshots, and maintenance while no user-facing latency or error rate moves. Check the alarm against the request path before treating it as an incident.
The alert is where the symptom surfaced. The cause usually lives one or two layers deeper. In most of the 20 incidents we analyzed, the initial alert pointed at the application or connection layer while the cause lived in the workload, the storage subsystem, or a schema change from weeks earlier.
Almost always slow queries, not too few connections. Concurrency in use equals arrival rate multiplied by mean hold time, so a query that goes from 4 milliseconds to 900 milliseconds multiplies held connections by more than 200 at the same traffic. Raising max connections pushes the failure into the database.
Check the layer where the symptom first appeared. If a Celery worker got evicted or an API returned 503, the database is often causing it while looking healthy itself. Trace the symptom back through the request path.
Read past the layer that alerted, name the specific dimension that saturated (not the service), and check whether the alarm reflects a real user-facing impact. Those three shifts consistently shortened the incidents we investigated.
Every one of the 20 investigations is browsable at sherlocks.ai/examples, with full evidence trails, signals pulled, and hypotheses ruled out.
See an AI SRE work a real incident
Book 30 minutes with our team and watch an investigation run on your own stack.
Book a demo →