Database incidents are usually reported by something other than the database. Workers get OOMKilled, an API times out, a connection pool empties, and the first alert names the application. Underneath, the cause sits in one of three layers that behave differently: the storage layer, where a volume hits a throughput or IOPS ceiling and every query pays for it; the query layer, where one plan regression after a migration changes the cost of a statement by two orders of magnitude; and the connection layer, where the pool is a symptom rather than a cause. The investigations here include a replica throttled on storage throughput, a pool exhausted by a missing index, and an alarm that was measuring something no user could feel.
gp3 data volume hit transient storage throttling after write throughput exceeded provisioned 750 MB/s (peaked 862.8 MB/s). Queue depth rose from ~11 to 17.89 and read latency doubled to 1,042 ms. Average IOPS stayed at 21% of provisioned 30,000, ruling out IOPS exhaustion.
Celery worker hit 91% of 1Gi memory limit as task backlog grew during a concurrent Aurora PostgreSQL load surge. Pod was evicted for Underutilized (not OOMKilled); surviving replica still at 87%. Thin headroom, not a crash.
Missing database index after schema migration caused 42-second queries (from 50ms). Connection pool across 3 app servers exhausted during morning peak, returning HTTP 503 for all users.
Short-lived write spike from upstream application caused a WriteIOPS alarm. Investigation confirmed no customer impact: transient burst from a batch job.
Connection pool exhaustion is the most commonly misread of the three, because the obvious reading (too few connections) is almost always wrong. Concurrency in use equals arrival rate multiplied by mean hold time, so a statement that goes from 4 ms to 900 ms multiplies held connections by more than two hundred at unchanged traffic. Check mean query duration and pg_stat_activity wait events before touching pool size: raising max connections on a plan regression moves the failure into the database. For storage, separate saturation from demand. Read await climbing while IOPS stays flat and throughput sits at the provisioned ceiling is a throttle, and it usually follows a change in row width or a sequential scan rather than a change in query rate. On a replica, confirm whether replay lag or client queries are consuming the budget, since they compete for the same volume. Finally, treat every database alarm as a claim to be checked against the request path. A WriteIOPS or CPU spike that does not correspond to a latency or error change is a workload event, and vacuum, backups, batch jobs, and analytics replicas all produce ones that look alarming and mean nothing.