How to fix database storage throttling (read latency high, IOPS fine)?

Every query against one instance is slow, the IOPS graph has enormous headroom, and nothing about the workload changed. Both observations are correct, and the contradiction is the diagnosis rather than a measurement error.

What does storage throttling look like from the database?#

It looks like every query getting uniformly slower, in proportion to how much data it reads, with no single statement standing out.

That uniformity is the tell, and it is what separates this from a plan regression. A regression makes one statement catastrophically slower and leaves the rest alone. A storage throttle taxes everything that touches the volume, so pg_stat_statements shows a broad rise in mean execution time rather than one outlier, and the queries that read the most pages degrade the most.

-- Wait events name the resource. Under a storage throttle the-- pile-up is on IO waits like DataFileRead, not on locks and-- not on ClientRead.SELECT count(*), wait_event_type, wait_eventFROM pg_stat_activityWHERE state = 'active'GROUP BY wait_event_type, wait_eventORDER BY count DESC; 

 count | wait_event_type | wait_event-------+-----------------+---------------    64 | IO              | DataFileRead     9 | IO              | WALSync     2 | Lock            | transactionid 

Backends stacked on IO / DataFileRead mean the database is waiting on the volume, not on itself.

Why does IOPS show headroom while the volume is throttled?#

Because IOPS and throughput are provisioned as two independent ceilings, and you can be pinned against one while idle on the other.

An operation is one I/O regardless of size. Read 8 KB and read 512 KB both count as a single operation, but the second moves 64 times more data. So a workload that shifts toward large sequential reads, a sequential scan, a wider row after a schema change, a nightly report, exhausts megabytes per second long before it exhausts operations per second. On a gp3 volume the default throughput is 125 MB/s no matter how much IOPS you bought, which is where over-provisioned IOPS quietly hides.

The result is a dashboard showing 21% utilisation on the graph everyone watches, while the volume is at 115% of a ceiling nobody graphed.

How do you confirm it is saturation rather than demand?#

Queue depth and latency together. Under a real throttle both climb, because requests are waiting for a resource that will not go faster. A volume that is merely busy shows high throughput with flat queue depth and steady latency.

# On the instance: await is service time plus queue time, and# aqu-sz is how deep the queue is. Both climbing while r/s is# flat is a throttle, not more work arriving.iostat -x 5 3 # %util saturating with low r/s and high rkB/s is the shape:# few operations, each large, against a throughput ceiling. 

Device   r/s     rkB/s      aqu-sz  r_await  %utilnvme1n1  2140    862,832    17.89   1042.0   99.8                 ^ 862.8 MB/s against 750 provisioned                          ^ was 11.0    ^ was 521 ms 

Then confirm what the volume is actually provisioned for, which the metrics alone never tell you.

# The ceilings, not the usage. A gp3 with 10,000 IOPS and# default throughput is an over-provisioned volume with an# under-provisioned dimension.aws ec2 describe-volumes --volume-ids vol-0a1b2c3d4e5f67890 \  --query 'Volumes[].{Type:VolumeType,Iops:Iops,ThroughputMBps:Throughput}' \  --output table 

Why do replicas throttle before the primary?#

Because a replica spends its storage budget twice, and the two halves compete.

A read replica applies the write stream from the primary and serves client queries from the same volume. Under write-heavy load, WAL replay consumes throughput that client queries then cannot have. The visible symptom is slow reads, but the cause may be replay, and adding another replica in that situation adds another consumer of the same primary's write stream rather than relieving anything.

-- Run on the replica. If replay is behind and the delay is-- growing, replay is spending the budget and read capacity is-- not the fix. If replay is current, client queries own it.SELECT pg_last_wal_receive_lsn()                      AS received,       pg_last_wal_replay_lsn()                       AS replayed,       pg_wal_lsn_diff(pg_last_wal_receive_lsn(),                       pg_last_wal_replay_lsn())      AS replay_bytes,       now() - pg_last_xact_replay_timestamp()        AS replay_delay; 

How do you fix database storage throttling?#

Raise the dimension that saturated, which on most volume types can be done live.

# Throughput and IOPS are modified independently. Raising IOPS# when throughput is the ceiling changes nothing, which is the# most common wasted remediation here.aws ec2 modify-volume --volume-id vol-0a1b2c3d4e5f67890 --throughput 1000 # The change is asynchronous. optimizing is not done, and the# incident is not closed until this reports completed.aws ec2 describe-volumes-modifications \  --volume-ids vol-0a1b2c3d4e5f67890 \  --query 'VolumesModifications[].{State:ModificationState,Progress:Progress}' 

Two ceilings sit above the volume and are worth knowing before you raise anything. gp3 throughput cannot exceed 0.25 MB/s per provisioned IOPS, so a very high throughput target may require raising IOPS purely as an enabler. And every instance type has its own EBS bandwidth limit, which a volume can never exceed no matter how it is provisioned.

Provisioning is the immediate fix, not the durable one. A throttle usually starts with a workload change, so the question worth answering afterwards is what began reading so much more data. A sequential scan introduced by a plan regression is a frequent answer, and it is cheaper to fix the plan than to buy throughput forever.

What does a real storage throttling incident look like?#

Sherlocks AI investigated one on a PostgreSQL replica, pg-slave-01. Its gp3 data volume breached provisioned throughput at 09:15:57 UTC, peaking at 862.8 MB/s against a provisioned 750 MB/s. VolumeQueueLength rose from about 11 to 17.89 and read latency doubled from 521 ms to 1,042 ms.

From the database side the signature was uniform: no single statement stood out, and the queries that read the most pages degraded the most. Average IOPS meanwhile stayed at 21% of provisioned for the whole event, so the graph the on-call engineer had open reported a volume with room to spare. The full writeup is in the EBS throughput throttle investigation.

This is the per-dimension trap, and it is an AWS pattern as much as a database one, so the EBS throughput throttling guide covers the same failure from the storage side. When slow storage shows up as pool pressure instead, connection pool exhaustion is the path it takes. More worked examples are on the database examples page, and the write-ahead log's own tuning surface is in the PostgreSQL WAL configuration documentation.

Database storage throttling FAQ#

How do I know storage is throttled and not just busy? Queue depth and latency rise together under a throttle. A busy volume shows high throughput with flat queue depth and steady latency.

Why is read latency high when IOPS is low? Because throughput is a separate ceiling. Large sequential reads exhaust megabytes per second long before they exhaust operations per second.

Why did raising IOPS not help? Because IOPS was not the constraint. On gp3, throughput is provisioned independently and defaults to 125 MB/s however much IOPS you buy.

Should I add a read replica when reads are slow? Not before checking replay lag. If WAL replay is consuming the volume budget, another replica adds a consumer of the same write stream rather than relieving it.

Can I raise throughput without downtime? Yes, modify-volume applies live, but it is asynchronous. Confirm with describe-volumes-modifications before closing the incident.

Could the instance be the bottleneck instead of the volume? Yes. Each instance type has its own EBS bandwidth ceiling, and no volume can exceed what its instance is rated to carry.