Alert info
- Severity
- Warning
- Detected by
- CloudWatch
- Alert
- pg-slave-01-data-disk-VolumeIOPSExceeded - Threshold Crossed: 1 datapoint [1.0] >= 1.0
- Time
- 2026-04-17 09:15:57 UTC
- Service
- pg-slave-01 (PostgreSQL replica ยท ap-south-1)
CloudWatch flagged a throttling event on the gp3 data volume of a PostgreSQL replica at 09:15:57Z. Investigate the cause.
gp3 data volume hit transient storage throttling after write throughput exceeded provisioned 750 MB/s (peaked 862.8 MB/s). Queue depth rose from ~11 to 17.89 and read latency doubled to 1,042 ms. Average IOPS stayed at 21% of provisioned 30,000, ruling out IOPS exhaustion.
- Write Peak
- 862.8 MB/s
- Queue Depth
- 17.89
- Read Latency
- 1,042 ms
Entities identified
Infrastructure
vol-0683a4215fecedb12
gp3 data disk
gp3 data volume for the PostgreSQL replica.
i-0f295fe40fde40ebb
EC2 host
EC2 instance hosting
pg-slave-01.
Service
read replicas
reporting / downstream
Reporting and downstream readers served off
pg-slave-01.
Data
pg-primary
PostgreSQL primary
PostgreSQL primary shipping WAL for replay.
pg-slave-01
PostgreSQL replica
PostgreSQL read replica in prod
ap-south-1.
External
CloudWatch
AWS/EBS metrics
CloudWatch metrics for AWS/EBS.
Gateway
VolumeIOPSExceeded
CloudWatch alarm
CloudWatch alarm tracking volume IOPS.
How they relate
The same relationships as text
- pg-primary WAL stream pg-slave-01
- pg-primary WAL stream pg-slave-01. Primary ships WAL; replica replays it as disk writes.
- pg-slave-01 writes / reads vol-0683a4215fecedb12. Replica I/O to the gp3 data disk.
- vol-0683a4215fecedb12 attached to i-0f295fe40fde40ebb
- vol-0683a4215fecedb12 backpressure pg-slave-01. Volume backpressure on the replica.
- CloudWatch scrapes vol-0683a4215fecedb12
- CloudWatch feeds VolumeIOPSExceeded. Metric values feed into the alarm evaluation.
- pg-slave-01 serves reads read replicas
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
IOPS exhaustion: the 30,000 provisioned IOPS limit was hitRuled out
Compute average IOPS from VolumeReadOps and VolumeWriteOps over the alarm window to check against the provisioned ceiling.
Total average IOPS peaked at ~6,263 (21% of 30,000) at 09:10-09:15Z. The IOPS ceiling was never approached. Despite the alarm name, IOPS is not the binding constraint.
5-minute average IOPS derived from raw ReadOps/WriteOps sums across the alarm window.
time ReadOps/300s avg read IOPS WriteOps/300s avg write IOPS
09:10Z 1,768,857 5,896 33,700 112
09:15Z 1,825,570 6,085 53,533 (peak) 178
total peak ~6,263 IOPS vs 30,000 provisioned (21%)
IOPS sat at 21% of the limit. This was not an IOPS exhaustion event.
Instance-level failure (crash, OOM, hardware fault) on the hostRuled out
Check EC2 instance state history and CPUUtilization for failures or saturation.
Instance stayed running throughout; no stop/terminate/reboot event. CPUUtilization peaked 86.98% at 09:05Z then eased to 67.53% by 09:15Z, a graceful recover, not a failure.
CPU rose with the write burst then receded; no instance lifecycle event.
time CPU % status
09:05Z 86.98% peak
09:10Z 85.30% elevated
09:15Z 67.53% easing
instance i-0f295fe40fde40ebb: running (no stop/terminate/reboot)
CPU eased on its own and the host never failed. High CPU shares the workload cause; it is not the driver.
Monitoring artifact: the IOPSExceeded datapoint is a false positiveRuled out
Look for independent corroboration of real storage contention using other metrics.
Queue depth and read latency both spiked independently in the same window, corroborating real EBS-side contention. A pure false positive is excluded. Ruled out (the metric API gap remains a noted caveat).
Queue and latency rose together during the alarm window, independent of the IOPSExceededCheck metric.
metric baseline at 09:15Z status
VolumeQueueLength 7.5 - 11.3 17.89 elevated
VolumeReadLatency 427 - 645 ms 1,042 ms ~2x baseline
IOPSExceededCheck (API returned no datapoints; alarm eval only)
Two independent metrics confirm real contention, so the alarm is not spurious.
gp3 write throughput burst exceeded the provisioned 750 MB/s ceilingConfirmed
Retrieve the volume configuration and compare VolumeWriteBytes against the provisioned throughput limit.
The volume is gp3 with 750 MB/s provisioned throughput. VolumeWriteBytes breached it at 08:55Z (812.9 MB/s) and peaked 862.8 MB/s at 09:00Z (115%), 10-15 minutes before the alarm. Throughput, not IOPS, is the binding constraint. Confirmed; trace the resulting backpressure next.
The data disk is gp3 with explicitly provisioned throughput and IOPS.
VolumeId vol-0683a4215fecedb12
VolumeType gp3
Size 6,100 GiB
Iops 30,000 (provisioned)
Throughput 750 MB/s (provisioned)
The 750 MB/s throughput ceiling is the constraint to test, not the 30,000 IOPS limit.
Write throughput breached and peaked above the gp3 ceiling minutes before the alarm.
time write MB/s vs 750 MB/s provisioned
08:15-08:50Z baseline < 750 ok
08:55Z 812.9 MB/s > 750 breach
09:00Z 862.8 MB/s > 750 peak (115%)
09:10Z 527.0 MB/s < 750 recovering
Throughput crossed the provisioned ceiling at 08:55-09:00Z, preceding the alarm by 10-15 minutes.
EBS throttled the volume, creating residual I/O backlogConfirmed
Query VolumeQueueLength and VolumeReadLatency to detect backlog buildup.
Queue depth rose from baseline ~7.5-11.3 to 17.31 (09:10Z) and 17.89 (09:15Z); read latency doubled from ~427-645 ms to 1,006-1,042 ms. The throttle produced real backpressure, which is what tripped VolumeIOPSExceededCheck=1. Confirmed; isolate the write driver next.
Queue length rose well above its baseline range during the alarm window.
time queue length status
08:15-08:50Z 7.5 - 11.3 baseline
09:10Z 17.31 elevated (+53% over baseline peak)
09:15Z 17.89 still elevated
Residual I/O backlog from the throttled throughput ceiling.
Read latency roughly doubled in lockstep with the queue rise.
time read latency status
08:15-08:50Z 427 - 645 ms baseline
09:10Z 1,006.78 ms ~2x baseline
09:15Z 1,042.12 ms ~2x baseline
Real EBS-side contention hitting PostgreSQL reads, correlated with the queue rise.
WAL replay from a primary-side workload burst drove the write spikeRoot cause
Analyze CPU metrics and PostgreSQL logs to identify the workload driving the replica write burst.
The replica's write spike is WAL replay of an unidentified primary-side burst: CPU peaked 86.98% at 09:05Z alongside the 862.8 MB/s write peak at 09:00Z, throughput crossed the gp3 750 MB/s ceiling, EBS throttled, and queue/latency backlog raised VolumeIOPSExceededCheck=1. The throttled throughput ceiling is the root cause; the burst was transient (throughput under limit by 09:10Z, host stayed running).
The CPU peak trails the write peak by minutes, consistent with workload-driven WAL replay rather than CPU starving I/O.
time CPU % write MB/s
09:00Z rising 862.8 (peak)
09:05Z 86.98% falling
09:10Z 85.30% 527.0
CPU follows the write burst, so a database workload (replayed WAL), not CPU, generated the I/O.
The alarm datapoint and the backlog align with the 08:55-09:00Z throughput breach.
08:55Z throughput breach begins (812.9 MB/s)
09:00Z peak 862.8 MB/s (115% of 750)
09:10Z IOPSExceededCheck=1; queue 17.31; latency 1,006 ms
09:15Z alarm OK -> ALARM; throughput already < 750 (transient)
The throttle and its backlog map cleanly onto the throughput breach, confirming throughput as the root constraint.
The exact PostgreSQL activity behind the burst (vacuum, checkpoint, backup, bulk load)Needs access
Attempt to retrieve PostgreSQL internal logs and maintenance schedules from both the primary and replica.
Outside the available telemetry. CloudWatch confirms the throughput throttle but cannot attribute the write burst to a specific database operation. The investigation labels the mechanism (WAL replay) rather than guessing the exact trigger.
No PostgreSQL metrics or logs were available to attribute the I/O spike to autovacuum, checkpoint, WAL replay of a bulk load, or a backup job.
The investigation stops at the EBS boundary and labels the mechanism. Needs privileged database access.
Replica index rebuild caused high write throughputRuled out
Check if a REINDEX operation was run on the replica.
Replicas are read-only and index builds happen on the primary and stream via WAL. The burst is WAL replay.
EBS volume bursting credits were exhaustedRuled out
Check the BurstBalance metric for the EBS volume.
gp3 volumes do not use burst credits like gp2, so this metric is not applicable and the issue is not burst exhaustion.
Final RCA
WAL replay from a primary-side workload burst drove the write spike
The replica's write spike is WAL replay of an unidentified primary-side burst: CPU peaked 86.98% at 09:05Z alongside the 862.8 MB/s write peak at 09:00Z, throughput crossed the gp3 750 MB/s ceiling, EBS throttled, and queue/latency backlog raised VolumeIOPSExceededCheck=1. The throttled throughput ceiling is the root cause; the burst was transient (throughput under limit by 09:10Z, host stayed running).
Remediation
# Check current gp3 throughput provisioning and evaluate increasing to 1000 MB/s
aws ec2 describe-volumes --volume-ids vol-0683a4215fecedb12 \
--query 'Volumes[0].{Type:VolumeType,Size:Size,Iops:Iops,Throughput:Throughput}'
# Review PostgreSQL checkpoint and WAL settings on the replica
psql -h pg-slave-01 -c "SELECT name, setting FROM pg_settings \
WHERE name IN ('checkpoint_timeout','checkpoint_completion_target',\
'max_wal_size','wal_level','wal_compression');"
# Check recent autovacuum activity that may have driven the write burst
psql -h pg-slave-01 -c "SELECT relname, last_autovacuum, last_autoanalyze \
FROM pg_stat_user_tables WHERE last_autovacuum > now() - interval '2 hours' \
ORDER BY last_autovacuum DESC LIMIT 10;"
# Add CloudWatch alarm on VolumeWriteBytes for earlier throughput breach detection
aws cloudwatch put-metric-alarm --alarm-name pg-slave-01-write-throughput \
--metric-name VolumeWriteBytes --namespace AWS/EBS \
--statistic Sum --period 300 --threshold 225000000000 \
--comparison-operator GreaterThanThreshold \
--dimensions Name=VolumeId,Value=vol-0683a4215fecedb12- Consider increasing gp3 throughput provisioning from 750 MB/s to 1,000 MB/s to accommodate write burst headroom on the replica.
- Add a CloudWatch alarm on VolumeWriteBytes throughput rate to detect breaches before they cascade into queue depth and latency degradation.
- Enable PostgreSQL logging for checkpoint and autovacuum activity (
log_checkpoints=on,log_autovacuum_min_duration=0) to attribute future I/O spikes to specific database operations. - Review backup/maintenance schedules on
pg-slave-01to identify whether scheduled jobs overlap with peak traffic and cause throughput contention. - Monitor VolumeQueueLength with a threshold alarm (e.g., > 15 for 2 of 3 datapoints) as an early indicator of EBS backpressure before IOPSExceeded triggers.
The throughput ceiling on this gp3 volume was set once at provisioning time and never revisited against the replica's actual write burst profile, so the fastest structural fix is a CloudWatch alarm on VolumeWriteBytes that fires before a breach turns into queue backlog, paired with headroom in the provisioned throughput itself. Because the exact database activity behind the burst was not attributable from CloudWatch alone, turning on checkpoint and autovacuum logging on the replica closes that gap for the next occurrence, and a VolumeQueueLength alarm gives an earlier, more specific signal than waiting for VolumeIOPSExceededCheck to trip.
Frequently asked questions
Why did a VolumeIOPSExceeded alarm fire when average IOPS was only 21% of provisioned?
The alarm name refers to the VolumeIOPSExceededCheck metric, which tracks gp3 throughput and IOPS throttling together, not IOPS alone. In this incident average IOPS peaked around 6,263, only 21% of the 30,000 provisioned, while write throughput hit 862.8 MB/s against a 750 MB/s provisioned ceiling. The throttle was on throughput, and the alarm name is misleading if you assume it only measures IOPS.
What is the difference between gp3 IOPS throttling and throughput throttling?
gp3 volumes provision IOPS and throughput as two independent ceilings. A volume can be nowhere near its IOPS limit while its throughput limit is breached, exactly as happened here: IOPS stayed at 21% of 30,000 provisioned while write throughput peaked at 862.8 MB/s against a 750 MB/s ceiling (115%). Checking only one dimension can miss a real throttle on the other.
How do you confirm an EBS throttling alarm reflects real contention rather than a monitoring glitch?
Corroborate the alarm with independent metrics. In this incident, VolumeQueueLength rose from a baseline of about 7.5 to 11.3 up to 17.89, and VolumeReadLatency roughly doubled from 427 to 645 ms baseline to 1,042 ms, both in the same window as the throughput breach. Two independent metrics moving together ruled out a pure false positive, even though the IOPSExceededCheck metric itself returned no datapoints when queried directly.
Does a gp3 throughput breach mean the EC2 instance is failing?
Not necessarily. In this incident the host instance stayed running throughout, with no stop, terminate, or reboot event. CPUUtilization peaked at 86.98% during the write burst and eased to 67.53% within ten minutes as throughput dropped back under the provisioned limit. The pressure was transient storage contention, not an instance-level failure.