AWS root cause analysis guides

AWS troubleshooting has a particular trap that Kubernetes does not. In Kubernetes, a failing pod usually tells you it is failing. In AWS, a managed service can be hard against a limit you cannot see, while every graph you are looking at shows headroom. The service is fine on the dimension you are watching and saturated on the one you are not.

That single idea explains most of the AWS incidents that waste hours. This page collects the common ones, with a guide for each, and a triage method that works across all of them.

Why do AWS incidents look fine on the dashboard?#

Because AWS limits are per-dimension, and dashboards default to the popular dimension, not the one that failed.

A gp3 EBS volume provisions IOPS and throughput as two separate ceilings. You can be at 100 percent of provisioned throughput and 21 percent of provisioned IOPS at the same moment. If your dashboard shows IOPS, which is the one most people watch, the volume looks healthy while it is throttling hard on megabytes per second. Same story on ElastiCache, where connection count and CPU are separate ceilings, so a cache can reject new connections while CPU and memory look calm. Same on anything with burst credits, which deplete on their own schedule regardless of current load.

So the first move in any AWS incident is not to ask whether the service is healthy. It is to ask which dimension saturated, because the service is almost always healthy on the dimensions you happened to be graphing.

The fastest way to prove this to yourself is to pull both EBS dimensions for the same window and convert them into the units each ceiling is actually provisioned in.

# gp3 provisions IOPS and throughput as two separate ceilings.# Pull both over the same window, at the same period, or the# comparison is meaningless.VOL=vol-0a1b2c3d4e5f67890WINDOW="--start-time 2026-04-17T09:00:00Z --end-time 2026-04-17T09:30:00Z --period 300" # Throughput: bytes summed per period, converted to MB/s.aws cloudwatch get-metric-statistics \  --namespace AWS/EBS --metric-name VolumeWriteBytes \  --dimensions Name=VolumeId,Value=$VOL \  --statistics Sum $WINDOW \  --query 'Datapoints[].[Timestamp,Sum]' --output text \  | sort | awk '{printf "%s  %8.1f MB/s\n", $1, $2/300/1024/1024}' # IOPS: operation counts over the same period.aws cloudwatch get-metric-statistics \  --namespace AWS/EBS --metric-name VolumeWriteOps \  --dimensions Name=VolumeId,Value=$VOL \  --statistics Sum $WINDOW \  --query 'Datapoints[].[Timestamp,Sum]' --output text \  | sort | awk '{printf "%s  %8.0f IOPS\n", $1, $2/300}' 

Put the two side by side against what the volume is actually provisioned for, and the answer stops being ambiguous. These are the real numbers from a throttled PostgreSQL replica:

DIMENSION      OBSERVED      PROVISIONED    UTILIOPS           2,140         10,000         21%      <- the graph everyone watchedThroughput     862.8 MB/s    750 MB/s       115%     <- the ceiling that broke VolumeQueueLength    11.0  ->  17.89Read latency        521ms  ->  1,042ms 

IOPS at 21 percent is what the on-call engineer saw. Throughput at 115 percent of provisioned is what actually happened. Queue depth and latency are the corroborating pair: they move together when a volume is genuinely throttled, and they stay flat when the metric is merely noisy.

# Confirm the volume's provisioned ceilings, which the metrics alone# never tell you. A gp3 defaults to 125 MB/s no matter how much# IOPS you bought, so this is where over-provisioned IOPS hides.aws ec2 describe-volumes --volume-ids $VOL \  --query 'Volumes[].{Type:VolumeType,SizeGiB:Size,Iops:Iops,ThroughputMBps:Throughput}' \  --output table 

How do you triage an AWS incident?#

The method is the same across all of these, and it comes down to four questions in order.

First, name the exact dimension that saturated, not the service. Every AWS limit is per-dimension: IOPS and throughput are provisioned separately on gp3, connection count and CPU are separate ceilings on ElastiCache, and burst credits deplete on their own schedule. A resource can be at 100 percent on one dimension and 20 percent on three others, so the volume looks fine is usually a statement about the wrong graph.

# List every alarm currently in ALARM state, with the metric and the# statistic each one is watching. The metric name is the dimension.aws cloudwatch describe-alarms --state-value ALARM \  --query 'MetricAlarms[].{Alarm:AlarmName,Metric:MetricName,Stat:Statistic,Period:Period,Threshold:Threshold}' \  --output table 

Second, decide whether the client or the service moved. Connection storms on a cache are frequently caused by pod churn, deploys, or an HPA cycling replicas. The cache is simply the place where a workload-side event first becomes visible, so a connection spike is often a symptom of something happening in your own application, not in the cache.

# The cache side: connections spiked, but is the cache itself unwell?redis-cli -h eaze-websocket-001.abc123.ng.0001.use1.cache.amazonaws.com \  INFO clients stats | grep -E 'connected_clients|rejected_connections|evicted_keys' # The client side, same window. If replicas churned here, the cache# is reporting your deploy back to you.kubectl get events -n prod --sort-by=.lastTimestamp \  --field-selector reason=ScalingReplicaSet | tail -20kubectl describe hpa eaze-channels -n prod | grep -A5 Events 

connected_clients:1788        # 280 -> 1,576.6 avg / 1,788 max in one 5-min bucketrejected_connections:0        # nothing was actually turned awayevicted_keys:0                # memory was never the constraint HPA eaze-channels  4 -> 17 replicas at 07:54Z 

Zero rejected connections and zero evictions, against a connection count that sextupled in one bucket, is the whole answer. The cache absorbed it. The HPA event at the same timestamp is the cause, and the fix belongs in the client's connection pooling, not in the cache's instance size.

Third, check whether the alarm reflects a customer-visible symptom at all. Compare the alarm period, statistic, and missing-data treatment against latency and error rates from the request path. A CloudWatch counter can spike from a snapshot, a maintenance operation, or a batch job while nothing a user touches slows down.

# What the alarm is actually measuring. A Sum over a 60s period and an# Average over 300s will disagree wildly about the same spike.aws cloudwatch describe-alarms --alarm-names content-writer-WriteIOPS-High \  --query 'MetricAlarms[].{Stat:Statistic,Period:Period,Threshold:Threshold,EvalPeriods:EvaluationPeriods,Missing:TreatMissingData}' # Now the request path over the identical window. This is the check# that decides whether it is an incident or a graph.aws cloudwatch get-metric-statistics \  --namespace AWS/RDS --metric-name ReadLatency \  --dimensions Name=DBClusterIdentifier,Value=content-writer \  --statistics Average --period 60 \  --start-time 2026-03-13T03:45:00Z --end-time 2026-03-13T04:15:00Z \  --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Average]' --output text 

If WriteIOPS breached 1,000 while latency and 5xx rate never moved, the batch job is not an incident. It is a static threshold set against a workload that has a bursty shape, and the correct fix is to the alarm, not to the database.

Fourth, read placement failures backwards. ECS and other scheduler failures make the most sense in reverse, from the scheduler event to the constraint that could not be satisfied at that moment, because AZ rebalancing can change the answer to a placement question that succeeded an hour earlier.

# The service event stream is where the real reason lives. The stopped# task only tells you that something failed, never which constraint.aws ecs describe-services --cluster prod --services url-swapping-service \  --query 'services[].events[:10].[createdAt,message]' --output text # Then the constraint the scheduler was trying to satisfy.aws ecs describe-task-definition --task-definition url-swapping-service \  --query 'taskDefinition.placementConstraints' 

TaskFailedToStart: MemberOf placement constraint unsatisfieddesired=2  running=2  pending=1     # churned, not downCPUUtilization  115.22% at 10:03Z   # a burst, not the cause 

The CPU burst is the loudest signal on the dashboard and it is not the cause. The placement constraint is, and it only became unsatisfiable because an AZ rebalance moved the candidate instances out from under it.

You can see these worked end to end, including the paths that were ruled out, on the AWS examples page. Several of these incidents also cross into Kubernetes, since an HPA cycling replicas or a pod eviction is often what a cache or database first reveals. For the AWS service mechanics behind these limits, the Amazon EBS, Amazon ElastiCache, and Amazon CloudWatch documentation are the primary references, alongside the PostgreSQL and Redis docs for the engines underneath.

Frequently asked questions#

Why does my AWS resource look healthy but still cause an incident?

Because AWS limits are per-dimension. The resource is likely saturated on a dimension your dashboard is not showing, like throughput instead of IOPS.

What does VolumeIOPSExceeded actually mean?

Often it is throughput, not IOPS. A gp3 volume can hit its provisioned megabytes-per-second ceiling while IOPS sits well below its limit.

Why did my ElastiCache connections spike when nothing changed?

Usually a workload-side event: pod churn, an HPA cycling replicas, or a client reconnect loop. The cache is where that first becomes visible, not the cause.

Is a CloudWatch alarm always a real incident?

No. Alarms can fire on batch jobs, snapshots, or maintenance while no user-facing latency or error rate moves. Always check the alarm against the request path.

How do I debug an ECS task placement failure?

Read it backwards, from the scheduler event to the constraint that could not be satisfied, since AZ rebalancing can change a placement answer that worked earlier.