Alert info
- Severity
- Critical
- Detected by
- Prometheus
- Alert
- PVC usage exceeded 90% threshold (prometheus-db volume)
- Time
- 2026-03-17 05:15 IST
- Service
- prometheus (monitor namespace)
PVC prometheus-db-prometheus-0 in the monitor namespace climbed past the 90% threshold, reaching 96.05% at alert time. Investigate the cause.
Prometheus PVC filled up due to TSDB block growth from increased metric cardinality under a 65-day retention window. Pod healthy: organic storage growth, not WAL bloat.
- PVC Usage
- 96%
- Retention
- 65 days
- Head Series
- 1.85M
Entities identified
Infrastructure
prometheus-prometheus
StatefulSet
StatefulSet managing the Prometheus pod.
gp3-xfs volume
/dev/xvdz
EBS gp3 volume backing the PVC.
retention 65d
--storage.tsdb.retention.time
Prometheus retention configuration.
Service
prometheus-prometheus-0
monitor ns
Prometheus pod.
Data
TSDB
head + blocks
Time-series database.
prometheus-db
PVC 350Gi
Persistent Volume Claim backing the database.
External
scrape targets
services + namespaces
Pods, services, and namespaces scraped by Prometheus.
How they relate
The same relationships as text
- scrape targets scraped by prometheus-prometheus-0
- scrape targets scraped by prometheus-prometheus-0. Scrape targets feed series into Prometheus.
- prometheus-prometheus-0 writes TSDB. Prometheus pod ingests samples and writes them to the TSDB head, then compacts into 2h blocks.
- prometheus-prometheus manages prometheus-prometheus-0
- TSDB persists blocks prometheus-db. TSDB blocks persist to the PVC.
- retention 65d bounds TSDB. Retention limits bound TSDB block storage.
- prometheus-db backed by gp3-xfs volume. PVC is bound to the gp3-xfs EBS volume mounted at /prometheus on
/dev/xvdz.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
WAL (write-ahead log) bloat is consuming the diskRuled out
Check the size of the /prometheus/wal directory to identify WAL bloat.
du shows /prometheus/wal = 1.4G only. WAL is bounded and truncating normally. Definitively excluded.
WAL is 1.4G against a 350Gi volume holding 320G used.
1.4G /prometheus/wal
(vs 319.7G total used on a 350Gi PVC)
WAL is a rounding error here, not the consumer. WAL bloat ruled out.
Pod instability / crashes causing re-ingestion or duplicate blocksRuled out
Query StatefulSet readiness and restart history for the Prometheus pod.
StatefulSet READY 1/1, AGE 45d, pod 2/2 Running with no restart events. Pod is stable.
No restarts on a 45-day-old StatefulSet; pod is Running and scheduled on the alerting node.
NAME READY AGE
prometheus-prometheus 1/1 45d
NAME READY STATUS NODE
prometheus-prometheus-0 2/2 Running ip-10-0-72-155.internal
No crash loop, no re-ingestion churn. Pod instability ruled out.
Stuck TSDB compaction leaving redundant blocks on diskRuled out
Examine prometheus_tsdb_head_series metrics to verify TSDB compaction health.
Head series rose 1.25M to 1.85M then dropped sharply to 1.29M, the signature of a healthy compaction / series lifecycle. Compaction is not stuck.
Head series peaked then dropped sharply as the head was flushed to a block.
time (UTC) head series observation
03:15 1.25M baseline
04:00 1.52M rising
04:45 1.85M peak cardinality
05:00 1.29M sharp drop (compaction)
The sharp drop after the peak is normal flush-to-block. Compaction is healthy, not stuck.
Organic TSDB block growth from rising metric cardinalityConfirmed
Analyze directory usage in /prometheus to identify large files or blocks.
TSDB blocks are the dominant consumer and newer blocks are markedly larger: ~9-10G (Feb) to 14-16G (Mar 12-17). Head series peaked at 1.85M. The disk is filling from more series per block, not from WAL, crashes, or stuck compaction. Confirmed; pin down the mechanism next.
Newer blocks are 60-70% larger than blocks from six weeks prior; blocks dominate the 320G used.
Filesystem Size Used Avail Use%
/dev/xvdz 350G 320G 30G 91%
9.2G 01JNAB2K... Feb 05-12
10.1G 01JP8R7X... Feb 19-26
13.2G 01JQ5X2M... Mar 05-12
14.8G 01JQHZ4N... Mar 12-15
16.1G 01JQR2W7... Mar 15-17 (current)
Block size is climbing week over week. The growth is organic ingestion, the dominant space consumer.
Rising cardinality inflates per-block size under a fixed 65-day retention windowConfirmed
Compare block growth rates with the configured retention period.
--storage.tsdb.retention.time=65d retains ~65 days of blocks. With head series up to a 1.85M peak, each block is proportionally larger, so 65 retained blocks now hold far more data than at baseline. df: 319.7G of 349.9G (96.05% at alert). Retention plus cardinality is the mechanism.
Retention is fixed at 65 days while per-block size grows with cardinality.
--storage.tsdb.retention.time=65d
--storage.tsdb.path=/prometheus
PVC usage: 91.29% (03:15) -> 96.05% (05:15)
df: 319.7G / 349.9G
At baseline cardinality 65d was fine; at 1.85M peak head series it retains too much. Reducing to 30-45d reclaims 20-30G.
Usage climbed steadily, confirming a genuine upward trend rather than a spike.
time (UTC) usage % status
03:15 91.29% warning
04:15 93.47% warning
05:00 95.38% critical
05:15 96.05% critical
Monotonic climb to 96.05%. The alert is real and the trend is upward.
New high-cardinality metrics / scrape targets added in the past 4-6 weeksRoot cause
Analyze series-by-metric and scrape_configs to determine if new metrics or targets were added.
The cardinality increase traces to scrape targets: new high-cardinality metrics / label dimensions and additional onboarded targets added series over the past 4-6 weeks, inflating every 2h block. Under 65-day retention this is the organic storage growth filling the PVC.
Top metrics by series count and active scrape targets grew versus the prior snapshot.
curl localhost:9090/api/v1/status/tsdb \
| jq '.data.seriesCountByMetricName[:20]'
head series: 1.25M baseline -> 1.85M peak
block size: ~9-10G (Feb) -> 16.1G (Mar 15-17)
More series per scrape -> larger blocks -> 65 days of them fill the 350Gi PVC. Organic growth from cardinality + retention.
Scrape interval was decreased, inflating block sizesRuled out
Review scrape configurations for changes to scrape_interval.
Scrape intervals remained at the default 15s; the growth was driven entirely by new series and targets.
Hidden deleted files are consuming disk spaceRuled out
Run lsof to identify open files that have been deleted but not released.
No large unlinked open files were found.
Final RCA
New high-cardinality metrics / scrape targets added in the past 4-6 weeks
The cardinality increase traces to scrape targets: new high-cardinality metrics / label dimensions and additional onboarded targets added series over the past 4-6 weeks, inflating every 2h block. Under 65-day retention this is the organic storage growth filling the PVC.
Remediation
# Update Prometheus CR or Helm values:
prometheus:
prometheusSpec:
retention: 30d
kubectl edit pvc prometheus-db-prometheus-0 -n monitor
# Set spec.resources.requests.storage: 500Gi
# Top 20 metrics by series count:
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:20]'- Reduce retention from 65d to 30-45d to free space immediately.
- Expand the PVC if retention must stay at 65d (increase to 500Gi or more).
- Identify and prune high-cardinality metrics contributing to block growth.
- Add recording rules or relabeling to drop unused high-cardinality labels before ingestion.
- Set up a PVC usage alert at 80% to catch growth earlier and allow time for remediation.
The PVC is filling because more series per scrape produces larger blocks under a retention window that was sized for an earlier, lower cardinality baseline, so the durable fix is bounding cardinality at the source rather than only reclaiming space after the fact. Pruning or relabeling the high-cardinality metrics identified via seriesCountByMetricName keeps future blocks close to their old size, and a PVC usage alert set around 80 percent gives time to act before the volume approaches full, unlike the 90 percent threshold that fired only once the trend was already critical. Retention and PVC size adjustments are useful levers for immediate headroom, but neither one stops the same growth from reappearing if the metric set keeps expanding unchecked.
Frequently asked questions
Why did a Prometheus PVC fill up even though the pod was healthy?
The StatefulSet was READY 1/1 with no restarts, so the fill was not a crash or instability issue. TSDB blocks were the dominant space consumer, with newer blocks 14 to 16 gigabytes versus 9 to 10 gigabytes six weeks earlier, so 65 days of retained blocks at higher cardinality simply took more disk than the same window did before.
How do you rule out WAL bloat as the cause of Prometheus disk growth?
Check the size of the write-ahead log directory directly. In this incident du showed /prometheus/wal at only 1.4 gigabytes against 319.7 gigabytes used on the volume, a rounding error next to the TSDB blocks. WAL bloat would show a large wal directory; here it was clearly bounded and truncating normally.
Does reducing Prometheus retention fix a PVC that is filling up from rising cardinality?
It reclaims space immediately but does not stop the underlying growth. Lowering retention from 65 days to 30 to 45 days frees the disk held by older blocks, while expanding the PVC or pruning the high-cardinality metrics identified via seriesCountByMetricName addresses the actual driver so the same fill does not recur.
Is stuck TSDB compaction a common cause of Prometheus disk growth?
In this incident it was ruled out. prometheus_tsdb_head_series rose from 1.25 million to a 1.85 million peak, then dropped sharply to 1.29 million, the signature of a normal flush to block rather than a stall. A stuck compaction would leave head series elevated instead of dropping after the peak.