How to fix a Kubernetes pod evicted for ephemeral storage?

How do you confirm it was ephemeral storage?#

Work from the pod outward. These three tell you whether this is an ephemeral-storage eviction, and whether the node or the workload is at fault.

# 1. What was the pod killed for?kubectl get pod <pod> -n <ns> -o jsonpath='{.status.reason}{"\t"}{.status.message}{"\n"}' # 2. Is the node still under pressure?kubectl get node <node> -o jsonpath='{range .status.conditions[?(@.type=="DiskPressure")]}{.status}{"\t"}{.message}{"\n"}{end}' # 3. Did the workload ever declare a budget?kubectl get pod <pod> -n <ns> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}' 

Command 1 returning Evicted with a message naming ephemeral-storage confirms the resource. Command 3 returning map[] is the finding that actually matters: no request means the scheduler never knew this pod needed disk.

One pod is rarely the whole story, though. The kubelet evicts in ranked order until the node is back under its threshold, so the pod that paged you is usually one of several:

# Every evicted pod in the cluster, with the reason the kubelet recordedkubectl get pods -A --field-selector=status.phase=Failed \  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,REASON:.status.reason,NODE:.spec.nodeName,MSG:.status.message' \  | grep Evicted 

Group that output by node. Several evictions on one node point at that node's disk; evictions scattered across many nodes point at a workload that fills whatever it lands on.

What does the eviction message actually say?#

kubectl describe pod on the evicted pod:

Status:       FailedReason:       EvictedMessage:      The node was low on resource: ephemeral-storage.              Threshold quantity: 2144337920, available: 522124Ki.              Container landing-page was using 1904164Ki, request is 0. Events:  Type     Reason   Age   From     Message  ----     ------   ----  ----     -------  Warning  Evicted  4m22s kubelet  The node was low on resource: ephemeral-storage.  Normal   Killing  4m22s kubelet  Stopping container landing-page 

Three numbers matter here. available: 522124Ki is roughly 510 Mi left against a Threshold quantity of 2.14 GB. was using 1904164Ki is the container's own consumption, about 1.8 GiB. And request is 0 is the root cause in one field: the scheduler placed this pod believing it needed no disk at all.

An evicted pod is not a crash. If you are separating this from a container that died on its own, the exit code is the tell, and the OOMKilled investigation walks the contrast: exit 137 with Reason: OOMKilled is the kernel stopping a process for memory, while an eviction is the kubelet removing a whole pod for disk.

Why did the kubelet evict the pod?#

The kubelet samples node filesystems every 10 seconds against configured thresholds. Crossing a hard threshold triggers immediate eviction with no grace period.

Read what your cluster actually uses, because managed distributions override the defaults:

# Effective kubelet config on a running node, thresholds includedkubectl get --raw "/api/v1/nodes/<node>/proxy/configz" | jq '.kubeletconfig.evictionHard' 

Before evicting anything, the kubelet tries to reclaim space by deleting unused images and dead containers. Eviction starts only when that is not enough. It then ranks pods: those exceeding their own ephemeral-storage request go first, and pods with no request are ranked by raw usage. That ranking is why the pod writing the most temp data is usually the one killed, even when a different pod triggered the fill.

How do you find what filled the disk?#

The eviction message names the victim, not the cause. To find the writer, get a shell onto the node:

kubectl debug node/<node> -it --image=busybox -- sh # Inside: the four directories that matter, largest firstdu -sh /host/var/lib/kubelet/pods/* 2>/dev/null | sort -rh | head -10du -sh /host/var/log/pods/* 2>/dev/null | sort -rh | head -10du -sh /host/var/lib/containerd 2>/dev/nulldf -h /host/var/lib/kubelet 

df disagreeing with du means deleted files are still held open by a running process, which is the signature of a log rotation that stopped. lsof +L1 names the holder.

For a per-pod view without leaving the API, the kubelet's Summary API reports actual ephemeral-storage bytes per pod:

kubectl get --raw "/api/v1/nodes/<node>/proxy/stats/summary" \  | jq -r '.pods[] | [.podRef.name, (.["ephemeral-storage"].usedBytes/1048576|floor)] | @tsv' \  | sort -k2 -rn | head 

That is the number the kubelet ranks on, so it is the one worth trusting.

What fills ephemeral storage in the first place?#

Ephemeral storage is local node disk, not a mounted volume. It covers the writable container layer, emptyDir volumes, and the stdout/stderr logs the kubelet captures. All of it disappears with the pod, and all of it is shared node disk, so one workload can starve its neighbours.

Four patterns cover nearly every incident:

• Application logs. A chatty service writing to stdout with no node-level rotation fills /var/log/pods steadily. Debug logging left on in production is the usual trigger.
• emptyDir with no sizeLimit. Scratch space, caches, and temp uploads write until the node disk is gone.
• The writable container layer. Downloads, unpacked archives, and generated files written to the container filesystem instead of a volume all count.
• No requests set, so the scheduler overcommits. The root of most incidents. Without a request the scheduler has no disk figure to bin-pack against and will happily place more pods than the disk can hold.

The pattern is not specific to one platform. Red Hat documents it for OpenShift in this eviction case, and it behaves identically on GKE, covered in this walkthrough of disk-pressure eviction. The mechanics are in the upstream node-pressure eviction reference.

How do you stop it happening again?#

Set ephemeral-storage on every container that writes local data. The request is what the scheduler bin-packs against; the limit is what evicts this pod alone instead of taking the node down with it.

resources:  requests:    memory: 256Mi    ephemeral-storage: 1Gi      # scheduler reserves this much node disk  limits:    memory: 512Mi    ephemeral-storage: 2Gi      # this pod is evicted at 2Gi, before the node fills 

Cap scratch space at the volume too, so an emptyDir cannot consume the node:

volumes:  - name: cache    emptyDir:      sizeLimit: 512Mi 

Then stop the growth at the source: turn off debug logging in production, confirm node-level log rotation is running, and write anything large or long-lived to a real volume. The full field reference is in the managing resources docs.

Verify the scheduler now sees the reservation:

kubectl describe node <node> | grep -A6 "Allocated resources" 

ephemeral-storage appearing with a non-zero request total means the overcommit is closed. To stop it reopening, LimitRange sets a namespace default so a workload shipped without requests still gets one:

apiVersion: v1kind: LimitRangemetadata:  name: ephemeral-storage-defaultsspec:  limits:    - type: Container      default:        ephemeral-storage: 2Gi      defaultRequest:        ephemeral-storage: 1Gi 

Why are replacement pods stuck in Pending?#

This is what turns an eviction into an outage. A node under DiskPressure carries a taint that repels new pods:

kubectl get node <node> -o jsonpath='{.spec.taints}' | jq 

[{"key":"node.kubernetes.io/disk-pressure","effect":"NoSchedule",  "timeAdded":"2026-04-17T09:15:57Z"}] 

While that taint is present the scheduler will not place replacements there, per taints and tolerations. If the rest of the cluster has no room, pods sit in Pending and the service stays degraded. kubectl describe pod <pending-pod> will say so directly: 0/6 nodes are available: 1 node(s) had untolerated taint.

On autoscaling clusters a provisioner like Karpenter adds capacity and the service recovers. That recovery is also the trap: the workload still has no requests, so it will overcommit the new node too. Autoscaling buys time, it does not fix the overcommit. Other failures wear the same disguise, where the cluster recovers and the cause survives — the CrashLoopBackOff investigation is one that ends in application code rather than infrastructure.

How do you catch it before the kubelet does?#

Alert on the trend, not the eviction. By the time a pod is evicted you are already down. The node exporter exposes the filesystem directly, and Prometheus can project the runway:

# Nodes whose kubelet filesystem will fill within 6 hourspredict_linear(node_filesystem_avail_bytes{mountpoint="/var/lib/kubelet"}[6h], 6*3600) < 0 # Anything already inside the eviction thresholdnode_filesystem_avail_bytes{mountpoint="/var/lib/kubelet"}  / node_filesystem_size_bytes{mountpoint="/var/lib/kubelet"} < 0.15 

Sizing matters too: on EKS the default node volume is frequently the constraint rather than the workload, covered in the Amazon EKS documentation.

What did a real ephemeral storage eviction look like?#

Sherlocks AI investigated this exact failure on a landing-page workload in production. Node ephemeral-storage fell to 510 Mi against a 2.14 GB threshold, the kubelet evicted the pod, and the resulting DiskPressure taint blocked replacements until Karpenter provisioned fresh capacity. The root cause was the empty resources block: no ephemeral-storage request, so the scheduler had been overcommitting that node's disk since the workload shipped.

The full write-up, including the signals checked and the hypotheses ruled out along the way, is the pod eviction root cause analysis. More worked incidents from the same cluster class are on the Kubernetes examples page.