How to fix EBS throughput throttling on a gp3 volume (VolumeIOPSExceeded)?

The alarm says IOPS. The graph says you have plenty of IOPS. Both are correct, and that is the whole problem. On gp3, IOPS and throughput are two separately provisioned ceilings, and VolumeIOPSExceeded is the alarm you get when either one throttles you.

What does VolumeIOPSExceeded actually mean?#

It means EBS throttled the volume. It does not tell you which limit did it.

A gp3 volume ships with a 3,000 IOPS and 125 MB/s baseline, and each can be provisioned upward on its own. Buying 10,000 IOPS does not raise throughput past 125 MB/s unless you also pay for throughput. That is where most of these incidents live: a volume with generous IOPS, default-ish throughput, and a write pattern made of large sequential blocks.

The arithmetic connecting them is just block size. Throughput equals IOPS multiplied by average I/O size. At a 16 KiB average, 10,000 IOPS needs 160 MB/s to complete. So a workload can be well inside its operation budget and hard against its byte budget at the same instant, and the volume gets throttled on the second one. The mechanics are set out in the Amazon EBS volume types documentation.

Why does the IOPS dashboard show headroom?#

Because almost every dashboard graphs VolumeReadOps and VolumeWriteOps, which are operation counts, and the ceiling you broke is measured in bytes.

Worse, the two default CloudWatch metrics are sums per period, not rates. A Sum of VolumeWriteOps over a 300 second period is a count, and reading it as IOPS without dividing by the period understates the real rate by 300x. Half the confusion in these incidents is a units problem before it is a capacity problem.

How do you confirm which ceiling you hit?#

Pull both dimensions over the same window and convert each into the units its ceiling is expressed in.

VOL=vol-0a1b2c3d4e5f67890WINDOW="--start-time 2026-04-17T09:00:00Z --end-time 2026-04-17T09:30:00Z --period 300" # Throughput: bytes summed per period, divided into MB/s.aws cloudwatch get-metric-statistics \  --namespace AWS/EBS --metric-name VolumeWriteBytes \  --dimensions Name=VolumeId,Value=$VOL \  --statistics Sum $WINDOW \  --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text \  | awk '{printf "%s  %8.1f MB/s\n", $1, $2/300/1024/1024}' # IOPS: operation counts over the same period, divided into ops/sec.aws cloudwatch get-metric-statistics \  --namespace AWS/EBS --metric-name VolumeWriteOps \  --dimensions Name=VolumeId,Value=$VOL \  --statistics Sum $WINDOW \  --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text \  | awk '{printf "%s  %8.0f IOPS\n", $1, $2/300}' 

Then read what the volume is actually provisioned for, which no metric will tell you:

aws ec2 describe-volumes --volume-ids $VOL \  --query 'Volumes[].{Type:VolumeType,SizeGiB:Size,Iops:Iops,ThroughputMBps:Throughput}' \  --output table 

Put the two side by side and the ambiguity disappears:

DIMENSION      OBSERVED      PROVISIONED    UTILIOPS           2,140         10,000         21%     <- the graph everyone watchedThroughput     862.8 MB/s    750 MB/s       115%    <- the ceiling that broke 

Why do queue depth and latency confirm it?#

Because a throttle is a queue, and a queue has a signature that a noisy metric does not.

When EBS throttles a volume, requests do not fail, they wait. VolumeQueueLength rises because operations are backing up, and read latency rises with it because each request now includes queue time. The two move together. If throughput looks high but queue depth and latency are flat, you are busy, not throttled.

# The corroborating pair. Both should climb together under a real throttle.for M in VolumeQueueLength VolumeTotalReadTime VolumeReadOps; do  echo "== $M"  aws cloudwatch get-metric-statistics \    --namespace AWS/EBS --metric-name $M \    --dimensions Name=VolumeId,Value=$VOL \    --statistics Average --period 300 \    --start-time 2026-04-17T09:00:00Z --end-time 2026-04-17T09:30:00Z \    --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Average]' --output textdone 

Read latency is not a metric of its own: it is VolumeTotalReadTime divided by VolumeReadOps. On a real throttle it roughly doubles.

VolumeQueueLength    11.0   ->  17.89Read latency         521ms  ->  1,042 ms 

How do you fix EBS throughput throttling?#

Match the fix to the dimension you confirmed, which is the entire point of the work above.

If throughput saturated, raise throughput. On gp3 this is a live modification with no downtime and no snapshot:

aws ec2 modify-volume --volume-id $VOL --throughput 1000 # Modification is asynchronous. Watch it to completion before you call it fixed.aws ec2 describe-volumes-modifications --volume-ids $VOL \  --query 'VolumesModifications[].{State:ModificationState,Progress:Progress,Throughput:TargetThroughput}' \  --output table 

gp3 accepts up to 1,000 MB/s, and throughput cannot exceed 0.25 MB/s per provisioned IOPS, so very high throughput requires IOPS to match even when IOPS is not your constraint.

If you are already at the volume ceiling, the constraint may not be the volume at all. Every instance type has its own EBS bandwidth limit, and a volume cannot exceed what its instance can carry:

aws ec2 describe-instance-types --instance-types m6i.xlarge \  --query 'InstanceTypes[].EbsInfo.EbsOptimizedInfo' --output table 

If the workload is the problem, reduce the bytes. Large sequential writes are usually a checkpoint, a vacuum, a backup, or a bulk load. Moving backups off the data volume, throttling a checkpointer, or spreading a bulk load over time all lower peak throughput without buying anything.

Then alarm on the right dimension, because the alarm that paged you did not name it:

# Throughput as a percentage of provisioned, using metric math rather than# a raw byte count nobody can eyeball against a ceiling.aws cloudwatch put-metric-alarm \  --alarm-name pg-slave-01-throughput-pct \  --threshold 85 --comparison-operator GreaterThanThreshold \  --evaluation-periods 2 \  --metrics '[    {"Id":"bytes","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"VolumeWriteBytes","Dimensions":[{"Name":"VolumeId","Value":"vol-0a1b2c3d4e5f67890"}]},"Period":300,"Stat":"Sum"},"ReturnData":false},    {"Id":"pct","Expression":"bytes/300/1048576/750*100","Label":"Throughput % of provisioned","ReturnData":true}  ]' 

What does a real EBS throttling incident look like?#

Sherlocks AI investigated one on a PostgreSQL replica. The gp3 data volume on pg-slave-01 breached its provisioned throughput at 09:15:57 UTC, peaking at 862.8 MB/s against a provisioned 750 MB/s. VolumeQueueLength rose from about 11 to 17.89 and read latency doubled to 1,042 ms.

The detail that made it clean: average IOPS stayed at 21 percent of provisioned for the entire event. Every IOPS dashboard showed a volume with enormous headroom, and every one of them was describing a dimension that was never the constraint. The alarm was named VolumeIOPSExceeded and the answer was throughput. The full writeup is in the EBS throughput throttle investigation.

This is the per-dimension trap in its purest form, and it recurs across AWS. The same shape appears when ElastiCache connections spike while cache CPU and memory look calm, and it is worth checking whether the alarm reflects user impact at all before treating it as an incident, which the CloudWatch false alarms guide covers. More worked examples are on the AWS examples page, and the storage engine's own view is in the PostgreSQL documentation.

EBS throughput throttling FAQ#

Does VolumeIOPSExceeded always mean I need more IOPS? No. On gp3 it fires for any throttle, and throughput is frequently the ceiling that broke while IOPS still shows headroom.

Why is my gp3 volume slow at only 21 percent IOPS? Because throughput is provisioned separately. Large sequential I/O exhausts megabytes per second long before it exhausts operations per second.

How do I tell throttling from a volume that is just busy? Queue depth and latency. Under a real throttle both climb together. A busy volume shows high throughput with flat queue depth.

Can I raise gp3 throughput without downtime? Yes. modify-volume applies live, but it is asynchronous, so confirm with describe-volumes-modifications before declaring the incident closed.

Why did raising IOPS not help? Because IOPS was never the constraint. Throughput on gp3 is capped independently and also cannot exceed 0.25 MB/s per provisioned IOPS.

Could the instance be the bottleneck rather than the volume? Yes. Each instance type has its own EBS bandwidth ceiling, and a volume can never exceed what its instance is rated to carry.