Databases root cause analysis guides
Database incidents are usually reported by the application rather than the database: a request timeout, an exhausted connection pool, a worker killed for memory. The pool is the symptom; a plan regression is the more common cause.
sre@prod-bastion ~ $ psql -f ~/oncall/activity.sql count | state | wait_event-------+---------------------+------------ 198 | active | ClientRead 2 | idle in transaction | # The pool is not too small. Concurrency is# arrival rate x hold time, and one statement# went from 4ms to 900ms after a migration.
In this series
(04 Guides)How do you trace an Aurora PostgreSQL load surge from a downstream symptom?
An Aurora load surge is usually reported by something else: a worker at 91 percent memory, a growing queue, an evicted pod. This guide covers reading the symptom back to the database, confirming it with pg_stat_activity and Performance Insights, and fixing the query rather than the worker.
How to fix a slow query after a schema migration (plan regression)?
One statement going from milliseconds to seconds after a migration is a plan regression, usually a missing index on a newly filtered column. This guide covers reading EXPLAIN ANALYZE for the regression, confirming it against the migration that caused it, and fixing it without a write lock.
How to fix database connection pool exhaustion (pool at 100%, HTTP 503)?
A connection pool at 100% is almost never a pool that is too small. This guide covers the arrival-rate-times-hold-time math that explains it, how to prove the pool is a symptom using pg_stat_activity, and why raising max connections makes the incident worse.
How to fix database storage throttling (read latency high, IOPS fine)?
Read latency climbing while IOPS shows headroom is a throughput throttle, not a workload spike. This guide covers separating saturation from demand using queue depth and await, the replica case where replay lag and client queries compete for one volume budget, and how to provision the dimension that actually saturated.
Each guide ends at a real incident. The matching Databases investigations show the same failures worked end to end, hypotheses and all.
Other stacks
- Kubernetes
- AWS
- Linux VMsSoon
See Sherlocks AI in action
Watch an AI SRE work a real incident from alert to root cause, on your stack, in 30 minutes.