How to fix a Linux VM disk filled by a dropped logrotate config (ENOSPC, no space left on device)?

The disk is full, the service is returning 500s, and nobody deployed anything. That combination is the signature of config drift, and the change that caused it is usually weeks behind you.

What does a full partition actually break?#

Every write that needs a new block, which is a much wider blast radius than logging.

When a filesystem hits capacity the kernel returns ENOSPC to any write that cannot be satisfied. Applications rarely handle that path well. Log writes fail, which is the visible symptom, but so do temp files, session files, upload staging, SQLite journals, and the socket and PID files a process expects to create on restart. A service that only wanted to append a log line ends up returning HTTP 500 for requests that never touched disk directly.

# Start here, not with du. df asks the kernel what the filesystem# reports; it is the number that decides whether a write succeeds.df -h /var # Inodes exhaust independently of blocks. A partition with free# space and zero free inodes fails writes the same way, and the# cause is different: many small files rather than a few huge ones.df -i /var 

Filesystem       Size  Used  Avail  Use%  Mounted on/dev/nvme0n1p1   100G   99G   800M   99%  /var 

Check inodes even when blocks look like the obvious answer. The two exhaust for opposite reasons, and running the wrong remediation against the wrong ceiling wastes the part of the incident where you still have room to act.

Why do du and df disagree, and what does it mean?#

Because they count different things, and the gap between them is a specific diagnosis rather than a rounding error.

df asks the filesystem how many blocks are allocated. du walks the directory tree and sums the files it can see. A file that has been deleted while a process still holds it open is gone from the tree but still holds its blocks, so df counts it and du cannot. That is the entire mechanism, and it splits the investigation cleanly in two.

# Same target, two different questions. Run them together and# compare, because either answer alone is ambiguous.df -h /var && du -sh /var 

If the two agree, the growth is real. Files exist, something wrote them, and the job is to find the directory and the writer. If they disagree, with df reporting more used than du can find, deleted files are still held open. That is the classic signature of a rotation that copied and unlinked a file without signalling the process to reopen it, or a truncate that never happened.

# +L1 means "link count less than 1", which is exactly the set of# deleted-but-still-open files. The holder is what you need: killing# or reloading it is what actually returns the blocks.sudo lsof +L1 /var 

Reaching for lsof +L1 when du and df agree is wasted time. It is a precise instrument for one branch of the fork, not a general disk-full tool.

How do you find what actually filled the disk?#

Descend by size rather than by suspicion. Two commands get you from a mount point to a single file.

# One level at a time. Sorting the whole tree at once buries the# answer in noise; this narrows to the one subtree that matters.sudo du -sh /var/* 2>/dev/null | sort -rh | head # Then the file itself, with its mtime. A file still being written# right now is a live writer, not an abandoned artifact.sudo ls -lh /var/log/order-service/ 

91G   /var/log4.2G  /var/lib1.8G  /var/cache800M  /var/tmp -rw-r--r-- 1 app app 87G Mar 30 11:42 app.log# no app.log.1, no app.log.2.gz 

The absent files are the finding. A directory under active rotation contains a generation of siblings: app.log.1, app.log.2.gz, and so on. A single enormous file with no siblings at all means rotation is not merely behind, it never ran for this service. That distinction matters because a slow rotation and an absent rotation have different causes and different fixes.

How do you prove rotation stopped rather than traffic grew?#

Compare the slope of the growth against the slope of the traffic. They separate the two cases immediately.

Unbounded accumulation from a missing reclaim mechanism is linear. Log volume per day stays flat because request rate stays flat, and the total climbs on a straight line. A genuine traffic or verbosity event is not linear: it is a step or a curve, and it shows up in request rate or in a deploy at the same timestamp.

req rate        ~2,800 req/min (steady, no deploys in window)daily log       ~6 GB/day87 GB / 6 GB    ~14.5 days of linear accumulation 

That arithmetic is the most useful thing on this page. Bytes used divided by bytes per day gives you a date, and a date gives you a window to search in change history. It is routinely more precise than anything in the logs, because the logs are the thing that stopped being managed.

Then confirm against logrotate's own record, which is authoritative about what it believes it is responsible for.

# Does a config exist for this service at all?ls /etc/logrotate.d/ # Has logrotate ever touched it? The status file is its memory.sudo grep order /var/lib/logrotate/status # Did logrotate run at all, and what did it do? A dry run shows# the set of files it considers in scope, without changing them.sudo logrotate -d /etc/logrotate.conf 2>&1 | grep -i 'considering\|log needs' 

$ ls /etc/logrotate.d/nginx  syslog  postgresql        # order-service MISSING $ sudo grep order /var/lib/logrotate/status(empty: never rotated) 

A healthy logrotate that rotated nginx, syslog and postgresql on schedule, with no entry at all for the service that filled the disk, is not a broken logrotate. It is a service that was never in scope. That rules out timer failures, permissions and lock contention in one command, and points the investigation at where the config was supposed to come from.

Why is the configuration management repository the first place to look?#

Because a VM is the output of its last converge, and the file that stopped existing was never in the application repository to begin with.

This is the structural reason VM incidents defeat the standard first move. Correlating with the most recent deploy searches the application repository over the last few hours. A dropped logrotate rule lives in Ansible, Chef, Puppet or Terraform, it shipped weeks ago, and it produced no visible change on the day it landed. The service kept running. Nothing rotated. The disk was the only thing keeping score.

# The template that should render the config. Its absence is the bug,# so search history rather than the working tree.git -C ~/ansible log --oneline --diff-filter=D --since='6 weeks ago' \  -- 'roles/**/logrotate*' # Then read the commit that removed it. Refactors are the usual# culprit: reorganising templates drops the ones nothing references.git -C ~/ansible show --stat a4f2c91 

Refactor commits are the ones to suspect. A commit titled "refactor log rotation to template-based config" reorganises files, and a template that no playbook explicitly references is easy to lose in the move. Review sees a tidier tree, CI sees green, and nothing in the pipeline asserts that every service still has a rotation rule.

Check the fleet as soon as you have the commit, because configuration management applies uniformly by design.

# The same converge ran everywhere. Peers are on the same curve,# just behind. This tells you how much runway you have.ansible app_servers -m shell -a 'df -h /var | tail -1; ls /etc/logrotate.d/' 

How do you reclaim the space without restarting the process?#

Truncate the file in place. Deleting it does not return the blocks while a process holds it open, which is how a disk-full incident becomes a disk-still-full incident.

# Truncate keeps the inode and the open file descriptor valid, so# the writer keeps appending to the same file at offset zero and# never needs a restart.sudo truncate -s 0 /var/log/order-service/app.log # Confirm the blocks actually came back before moving on.df -h /var 

Do not use rm here. Unlinking a file that a running process holds open removes the directory entry and leaves the blocks allocated until that process exits or closes the descriptor. You get no space back, du and df now disagree, and you have converted a clear problem into the ambiguous one from the section above.

With the immediate pressure gone, restore the rotation rule and then fix the source so the next converge does not undo you.

# The rule itself. size 1G is the important line: it bounds the# damage from a future gap instead of relying on daily alone.sudo tee /etc/logrotate.d/order-service > /dev/null << 'EOF'/var/log/order-service/*.log {    daily    rotate 7    compress    missingok    notifempty    size 1G    postrotate        systemctl reload order-service    endscript}EOF # Prove it before trusting it.sudo logrotate -d /etc/logrotate.d/order-service # Then fix the actual source, or the next converge removes it again.git -C ~/ansible revert a4f2c91 

Restoring the file on the host without restoring the template is the trap. The host is downstream. The next converge is authoritative, and it will happily delete your hand-written config and reset the clock.

What does a real disk full incident look like?#

Sherlocks AI investigated one on order-service at 11:42 IST on 30 March 2026. Prometheus Node Exporter paged on /var at 99.2 percent on app-server-02, with 800 MB free on a 100 GB volume, and the service was returning HTTP 500s. 342 orders were affected.

Three hypotheses were ruled out before the cause surfaced. A traffic or log surge was rejected because growth was a flat 6 GB per day against a steady 2,800 requests per minute, which is linear accumulation rather than a spike. Another process filling /var was rejected from du: /var/lib, /var/cache and /var/tmp came to roughly 7 GB combined while /var/log held 91 GB. A runaway error loop or verbosity change was rejected because there were no deploys in the window and no log-level change, and the lines were routine request entries averaging about 35 bytes.

What confirmed it was the absence of siblings: app.log was a single 87 GB file with no app.log.1 and no .gz archives, logrotate had run at 03:00 and rotated nginx, syslog and postgresql, and /var/lib/logrotate/status had no entry for the service at all. Git blame on the Ansible repository found commit a4f2c91 from 14 days earlier, titled "Refactor log rotation to template-based config", which removed roles/app/templates/logrotate-order-service.j2 with nothing restoring it. The growth math agreed independently: 87 GB at 6 GB per day is about 14.5 days, matching 16 March to 30 March. The same converge had dropped the rule fleet-wide, leaving app-server-01 at 78 percent and app-server-03 at 71 percent, three to five days behind on the identical curve. The full writeup is in the disk full log rotation investigation.

The prevention is a CI check that diffs the set of services against the set of logrotate configs in the configuration management repo, which would have failed at merge time instead of fourteen days later. The sibling failure mode on these hosts is a slow memory climb ending at the kernel, covered in JVM memory leaks and Linux OOM kills. More worked examples are on the Linux VMs examples page, and the systemd journal documentation covers the equivalent controls when journald rather than logrotate owns retention.

Disk full FAQ#

Why did deleting the log file not free any space? Because a process still holds the file open. Unlinking removes the directory entry but not the blocks, which stay allocated until the descriptor closes. Use truncate -s 0 instead, which keeps the descriptor valid and returns the space immediately.

What does it mean when du and df disagree? df counts allocated blocks, du sums files it can see in the tree. When df reports more used than du finds, deleted files are still held open by a running process. lsof +L1 names the holder.

How do I date the change that caused a slow disk fill? Divide the bytes consumed by the observed bytes per day. Linear growth means the accumulation started when the reclaim mechanism stopped, so the quotient is the number of days back to search in change history.

Why is logrotate running fine but not rotating my service? Because it has no config for that service. A missing file in /etc/logrotate.d/ and no entry in /var/lib/logrotate/status means the service was never in scope. The timer, permissions and locks are all irrelevant in that case.

Should I look in the application repository or the configuration management repository? The configuration management repository, first. Rotation rules, sysctls and unit files live there, and a VM is the output of its last converge. An application deploy will not look suspicious because the application did not change.

How do I stop this recurring after I restore the config? Fix the template in the configuration management repo rather than only the host, since the next converge overwrites hand edits. Add a size directive so rotation is not purely time-based, and alert on disk at 80 percent so a future gap surfaces with days of runway rather than minutes.