Alert info
- Severity
- Critical
- Detected by
- Prometheus Node Exporter + PagerDuty
- Alert
- Filesystem /var usage exceeded 95% on app-server-02
- Time
- 2026-03-30 11:42 IST
- Service
- order-service (app-server-02)
/var on app-server-02 hit 99.2% with only 800 MB free. Investigate the cause.
Ansible playbook refactor accidentally removed logrotate config. 14 days of unrotated logs (87 GB) filled the 100 GB /var volume, causing ENOSPC errors and HTTP 500s.
- /var Usage
- 99.2%
- Log File
- 87 GB
- Affected Orders
- 342
Entities identified
Infrastructure
app-server-02
RHEL 8.9 host
VM hosting order-service.
logrotate
nightly 03:00
Log rotation service.
log-mgmt playbook
Ansible
Ansible playbook managing logs.
app-server-01 / 03
same playbook
Peer application servers.
Service
order-service
Node.js 20
Revenue API service.
Data
/var volume
100 GB EBS gp3
/var filesystem.
app.log
/var/log/order-service
Application log file.
Gateway
Node Exporter
Prometheus + PagerDuty
Prometheus monitoring.
How they relate
The same relationships as text
- order-service runs on app-server-02
- order-service writes app.log. order-service writes to
app.log. - app.log fills /var volume.
app.logis stored on the /var volume. - /var volume ENOSPC order-service. Volume pressure affects order-service.
- logrotate should rotate app.log.
logrotatemanagesapp.log. - log-mgmt playbook dropped config logrotate. Ansible configures
logrotate. - Node Exporter monitors /var volume. Prometheus monitors volume usage.
- log-mgmt playbook same run app-server-01 / 03. Ansible configures peer hosts.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
A sudden traffic or log surge generated an abnormal log volumeRuled out
Compare request rate and log growth metrics to check for an anomalous burst.
Growth is a flat ~6 GB/day at steady 2,800 req/min: 87 GB over ~14 days is linear, not a spike. Peer hosts show proportionally smaller logs matching lower traffic. No surge.
Daily log volume tracks steady request rate, no anomalous burst.
req rate ~2,800 req/min (steady)
daily log ~6 GB/day
87 GB / 6 GB ~14.5 days (linear)
Linear 14-day accumulation, not a surge. The cause is structural, not a traffic event.
Another process (package cache, core dumps, images) filled /varRuled out
Analyze directory sizes within /var using du to identify the largest consumer.
/var/lib 4.2 GB, /var/cache 1.8 GB, /var/tmp 800 MB: combined non-log usage is ~7 GB. /var/log holds 91 GB. The fill is logs, not another process.
Per-subtree disk usage across the /var volume.
91G /var/log
4.2G /var/lib
1.8G /var/cache
800M /var/tmp
/var/log dominates at 91 GB; everything else is ~7 GB combined. The fill vector is logs.
A runaway error loop or verbosity change inflated the logsRuled out
Check deployment history and app.log contents for verbosity changes or error-spam.
No deploy or log-level change in the window. ~6 GB/day matches normal INFO-level request logging at 2,800 req/min, and the accumulation is a smooth 14-day line, not the steep curve a runaway loop would produce.
No config change to log level; log lines are routine request entries, not repeated errors.
deploys in window none
log level INFO (unchanged)
line type request access lines (~35 B/line avg)
Steady INFO volume over 14 days, not an error storm or verbosity bump.
order-service log was never rotated, so a single file grew unboundedConfirmed
Audit logrotate configurations and status files to confirm rotation for the service.
app.log is one 87 GB file with no app.log.1 / .gz siblings. logrotate ran 03:00 today and rotated nginx, syslog, postgresql but has no order-service config and no status entry. Rotation was never happening. Confirmed; find why the config is gone.
No order-service config, no status entry, no rotated files.
$ ls /etc/logrotate.d/
nginx syslog postgresql # order-service MISSING
$ grep order /var/lib/logrotate/status
(empty: never rotated)
$ ls -lh /var/log/order-service/
-rw-r--r-- 1 app app 87G Mar 30 11:42 app.log
# no app.log.1, no app.log.2.gz
order-service is outside logrotate's scope entirely; the single file grew unbounded.
An Ansible playbook refactor dropped the logrotate template 14 days agoRoot cause
Examine version control history for Ansible playbooks managing logrotate configuration.
Commit a4f2c91 (Mar 16) 'Refactor log rotation to template-based config' removed roles/app/templates/logrotate-order-service.j2 and nothing restored it. 87 GB / ~6 GB per day = ~14.5 days, matching Mar 16 to Mar 30. The same run dropped the config on app-server-01 (78%) and app-server-03 (71%).
The commit that removed the order-service logrotate template, and the date.
commit a4f2c91 (Mar 16, ~14 days ago)
"Refactor log rotation to template-based config"
- removed: roles/app/templates/logrotate-order-service.j2
(no later commit re-adds it)
The refactor silently deleted the template; logrotate has skipped order-service ever since.
Accumulation duration and peer-host usage line up with the same Ansible run.
87 GB / ~6 GB/day ~14.5 days (Mar 16 -> Mar 30)
app-server-02 99.2% 87 GB full (incident)
app-server-01 78% 62 GB ~3-4 days
app-server-03 71% 54 GB ~5 days
Timeline and fleet-wide spread both trace to commit a4f2c91. Single, confirmed origin.
The logrotate process is failing or not runningRuled out
Check the status and logs of the logrotate systemd timer/service.
logrotate ran successfully at its scheduled times; the issue is a missing configuration file for this specific service.
Final RCA
An Ansible playbook refactor dropped the logrotate template 14 days ago
Commit a4f2c91 (Mar 16) 'Refactor log rotation to template-based config' removed roles/app/templates/logrotate-order-service.j2 and nothing restored it. 87 GB / ~6 GB per day = ~14.5 days, matching Mar 16 to Mar 30. The same run dropped the config on app-server-01 (78%) and app-server-03 (71%).
Remediation
# Emergency: Truncate the oversized log file (keeps file handle open)
ssh app-server-02 'truncate -s 0 /var/log/order-service/app.log'
# Restore logrotate config on ALL app servers
cat > /etc/logrotate.d/order-service << 'EOF'
/var/log/order-service/*.log {
daily
rotate 7
compress
missingok
notifempty
size 1G
postrotate
systemctl reload order-service
endscript
}
EOF
# Fix the Ansible playbook: restore the template
git revert a4f2c91 # or re-add logrotate-order-service.j2- Restore
logrotateconfig on all three app servers immediately (app-server-01and 03 will hit the same issue within days). - Fix the Ansible playbook to include the order-service
logrotatetemplate. - Add a CI check that verifies all services have corresponding
logrotateconfigs in the Ansible repo. - Add disk usage alerts at 80% (warning) and 90% (critical) with enough lead time to act before ENOSPC.
- Consider shipping logs to a centralized system (ELK/Loki) and reducing local retention to 3 days.
- Set filesystem reservation (tune2fs -m 5%) to keep 5% reserved for root, preventing complete disk exhaustion.
The Ansible refactor that dropped the logrotate template passed review because nothing in the pipeline checked that every service still had a rotation rule after the template reorganization. A CI check that diffs the set of services against the set of logrotate configs in the Ansible repo would have caught the gap at merge time instead of fourteen days later. Layering an earlier disk alert at 80 percent on top of that check means even a future gap surfaces with days of runway instead of minutes, and shipping logs to a centralized store with a short local retention window shrinks how much any single missed rotation can cost.
Frequently asked questions
Why did /var fill up on app-server-02?
The order-service application log grew to 87 GB as a single unrotated file because the logrotate configuration for that service was missing from /etc/logrotate.d/. At about 6 GB per day, 14 days of unchecked growth on a 100 GB /var volume filled the disk to 99.2 percent and left the application unable to write logs or temp files, returning ENOSPC errors and HTTP 500.
How was the missing logrotate config traced back to its cause?
Git blame on the Ansible repository traced the missing config to commit a4f2c91, made 14 days earlier and titled Refactor log rotation to template-based config. That commit removed roles/app/templates/logrotate-order-service.j2 and nothing restored it, so logrotate ran nightly on schedule but had no rule for order-service and silently skipped it every night since.
Were other servers affected by the same missing config?
Yes. The same Ansible run that dropped the config applied it fleet-wide, so app-server-01 and app-server-03 carried the identical missing logrotate rule. At the time of the incident they sat at 78 percent (62 GB of logs) and 71 percent (54 GB of logs) respectively, roughly three to five days behind app-server-02 on the same unbounded growth curve.
What is the fix for a dropped logrotate configuration like this?
The immediate fix was truncating the 87 GB file in place to free disk space without restarting the process, then restoring the logrotate configuration on all three app servers and reverting the Ansible commit that removed the template. A CI check that verifies every service has a corresponding logrotate config would catch this class of regression before it reaches production.