Alert info
- Severity
- Warning
- Detected by
- Kubernetes / prod-india-exchange
- Alert
- Pod landing-page-79789c969c-jb7x4 was Evicted
- Time
- 2026-04-17 11:52:41 UTC
- Service
- landing-page (prod-india-exchange)
Pod evicted from node ip-10-50-15-152 with low ephemeral-storage message. Investigate the cause.
Pod evicted by kubelet after node ran out of ephemeral-storage (510 Mi available vs 2.14 GB threshold). DiskPressure taint blocked replacements; Karpenter provisioned new capacity. Missing ephemeral-storage requests allowed silent overcommit.
- Disk Available
- 510 Mi
- DiskPressure
- True
- Recovery
- Karpenter autoscale
Entities identified
Infrastructure
prod-india-exchange
EKS cluster
EKS cluster in
ap-northeast-1. Karpenter-managed nodes. Hosts the landing-page workload.ip-10-50-15-152
ap-northeast-1
Pressured node hosting the evicted pod.
kubelet
node agent
Node agent managing pods and node state.
landing-page deploy
pod spec
Deployment spec for landing-page.
Karpenter
autoscaler
Cluster autoscaler.
Service
landing-page-...-jb7x4
evicted pod
Pod experiencing eviction symptoms.
landing-page-79789c969c
current ReplicaSet
Current ReplicaSet for the workload.
landing-page-746746859f
prior ReplicaSet
Older ReplicaSet on other nodes.
Data
ephemeral-storage
node local disk
Node local disk.
How they relate
The same relationships as text
- prod-india-exchange runs landing-page-79789c969c
- prod-india-exchange runs landing-page-746746859f
- landing-page-79789c969c owns landing-page-...-jb7x4
- landing-page deploy defines landing-page-79789c969c. Deployment spec rolls out the current ReplicaSet with no ephemeral-storage request/limit.
- landing-page-79789c969c scheduled onto ip-10-50-15-152. Scheduled pods onto nodes.
- landing-page-...-jb7x4 runs on ip-10-50-15-152
- ip-10-50-15-152 backs ephemeral-storage. Node provides local disk storage.
- kubelet monitors / taints ip-10-50-15-152. Monitors node conditions and sets taints.
- kubelet evicts landing-page-...-jb7x4. Evicts pods based on resource pressure.
- landing-page deploy no request/limit ephemeral-storage. Defines storage requirements.
- Karpenter provisions capacity prod-india-exchange. Provisions additional capacity.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Application crash or bad deploy failed the podRuled out
Read pod describe to determine if the terminal reason is an internal error or external eviction.
Reason is Evicted, an external kubelet action, not an in-process failure. No Error, CrashLoopBackOff, or OOMKilled.
Status=Failed, Reason=Evicted, message names ephemeral-storage.
$ kubectl describe pod landing-page-79789c969c-jb7x4 -n landing-page
Status: Failed
Reason: Evicted
Message: The node was low on resource: ephemeral-storage.
Container landing-page was using 68Ki, request is 0,
which exceeds its limit of 0.
Eviction is kubelet-initiated, not an app crash. H1 ruled out.
Memory / CPU / PID pressure caused the failureRuled out
Check node conditions for MemoryPressure, PIDPressure, or DiskPressure.
MemoryPressure=False, PIDPressure=False, CPU allocatable 15,890m. Only DiskPressure=True. The failure domain is ephemeral-storage alone.
Only DiskPressure is True; memory, PID, and CPU are healthy.
Conditions:
MemoryPressure False
DiskPressure True <- only pressure
PIDPressure False
Ready True
Allocatable cpu: 15,890m
Single failure domain: disk. Not memory/CPU/PID. H2 ruled out.
False positive: pod was healthy, alert misfiredRuled out
Check if replacement pods also failed and if a node taint was applied.
Real eviction: DiskPressure=True, disk-pressure:NoSchedule taint set, and replacement pods independently failed image extraction with "no space left on device". Not a monitoring artifact.
New pods could not extract image layers; disk was already full.
Warning Failed ErrImagePull
Warning Failed failed to extract layer: ...
no space left on device
Disk exhaustion is independently confirmed. H3 ruled out.
Node ephemeral-storage exhaustion triggered kubelet hard evictionConfirmed
Compare available ephemeral-storage against the kubelet hard eviction threshold.
Available was 510,792 Ki (~499 MB) against a 2,139,512,454 byte (~2.14 GB) hard threshold. EvictionThresholdMet fired and kubelet reclaimed ephemeral-storage by evicting jb7x4, then tainted the node. Confirmed; the open question is what consumed the disk and why the scheduler allowed it.
Available ephemeral-storage well below the hard eviction threshold.
eviction threshold 2,139,512,454 bytes (~2.14 GB)
available at evict 510,792 Ki (~499 MB)
node allocatable ~18 GB ephemeral-storage
Hard threshold crossed. kubelet evicts to reclaim disk.
DiskPressure=True and a NoSchedule taint applied during pressure handling.
DiskPressure True
Taints: node.kubernetes.io/disk-pressure:NoSchedule
Event: Normal EvictionThresholdMet
Attempting to reclaim ephemeral-storage
kubelet pressure handling is active. Confirmed root mechanism.
Disk consumed by image cache / logs / co-located pods, not landing-page itselfConfirmed
Analyze node disk usage to determine what consumed the ephemeral storage.
The landing-page container used only 68 Ki; ~17.5 GB was consumed by other sources (image cache, container logs, co-located pods, kubelet overhead). Replacement pods hit "no space left on device" extracting image layers, pointing at disk-level fill rather than this workload.
Huge gap between this container's usage and total disk consumed.
landing-page usage at evict 68 Ki
node allocatable ~18 GB
available at evict ~499 MB
gap (other sources) ~17.5 GB
Bulk of disk is consumed off-workload: image cache, logs, co-located pods.
Unset ephemeral-storage requests let the scheduler silently overcommit the nodeRoot cause
Review the deployment pod spec for ephemeral-storage configurations.
request=0 and limit=0 on the landing-page container. With no reservation signal the scheduler treats node disk as infinite and keeps packing workloads until kubelet hard-eviction thresholds fire. This configuration gap is the root cause that allowed the disk to fill silently and the eviction to occur. Recovery came only after Karpenter provisioned new capacity.
No ephemeral-storage request or limit on the container.
Container landing-page:
ephemeral-storage request: 0 <- unset
ephemeral-storage limit: 0 <- unset
Impact: scheduler treats disk as infinite -> overcommit
-> kubelet evicts at hard threshold
Missing requests removed the scheduler's disk-accounting. Root cause.
Replacements only scheduled after new node capacity arrived.
11:52Z jb7x4 evicted (DiskPressure)
11:52-53Z 4pvpn/d5xcg FailedScheduling (taint, insufficient)
11:55Z Karpenter provisioned new node
11:57Z replacements scheduled, entering Running
Tainted node blocked placement; Karpenter capacity restored service.
Node logs rotation failed causing disk fillRuled out
Check the node's log rotation service (logrotate).
Log rotation was functioning correctly; the disk filled from legitimate image cache and pod data accumulation.
Final RCA
Unset ephemeral-storage requests let the scheduler silently overcommit the node
request=0 and limit=0 on the landing-page container. With no reservation signal the scheduler treats node disk as infinite and keeps packing workloads until kubelet hard-eviction thresholds fire. This configuration gap is the root cause that allowed the disk to fill silently and the eviction to occur. Recovery came only after Karpenter provisioned new capacity.
Remediation
# Add ephemeral-storage requests and limits to prevent silent overcommit
kubectl set resources deploy/landing-page -n landing-page \
--requests=ephemeral-storage=500Mi --limits=ephemeral-storage=1Gi
# Check what else was consuming disk on the pressured node
kubectl get pods --field-selector spec.nodeName=ip-10-50-15-152.ap-northeast-1.compute.internal \
-A -o wide
# Add node disk usage monitoring alert
# PromQL: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15- Set ephemeral-storage requests (e.g., 500Mi) and limits (e.g., 1Gi) on all deployments so the scheduler can account for disk usage and avoid overcommitting nodes.
- Add a Prometheus alert on node ephemeral-storage usage (e.g., > 80%) to detect disk pressure before kubelet eviction thresholds are reached.
- Audit co-located workloads on the affected node to identify the primary disk consumer (image cache, container logs, temporary files).
- Configure kubelet garbage collection thresholds for unused images and dead containers to free disk space proactively.
- Review Karpenter node provisioning to ensure nodes have sufficient ephemeral-storage for the expected workload mix, including image pull headroom.
Ephemeral-storage is the resource dimension teams most often forget to request, because a pod can run for months before a co-located workload fills the node's local disk. Setting a request and limit on every deployment gives the scheduler a real accounting signal instead of treating disk as infinite, so overcommit stops before it starts. Pairing that with a node-level disk usage alert well below the kubelet hard eviction threshold, and kubelet image garbage collection watermarks, turns a silent multi-week fill into a routine capacity ticket instead of an eviction and a scramble to autoscale replacement nodes.
Frequently asked questions
What does 'Evicted' mean for a Kubernetes pod?
Evicted means the kubelet, not the application, terminated the pod because the node ran low on a resource it monitors, such as memory, PID count, or disk. In this incident the pod's terminal message read "The node was low on resource: ephemeral-storage", and node conditions confirmed DiskPressure=True while MemoryPressure and PIDPressure were both False, isolating the failure domain to disk.
Why does an unset ephemeral-storage request cause node overcommit?
When a container spec leaves ephemeral-storage request and limit at 0, the scheduler has no signal to account for disk usage and keeps placing pods on the node as if storage were unlimited. In this incident the deployment had request=0 and limit=0, so the node kept accumulating image cache, logs, and co-located pod data until available space fell to 510,792 Ki, well under the 2.14 GB hard eviction threshold.
Why did replacement pods also fail after the eviction?
The node was tainted disk-pressure:NoSchedule after the eviction, and replacement pods that did land on the same node failed image extraction with "no space left on device" because the disk was already exhausted. Recovery only completed once the Karpenter autoscaler provisioned new node capacity and the replacement pods were rescheduled there.
How do you prevent ephemeral-storage exhaustion from evicting pods again?
Set explicit ephemeral-storage requests and limits on every deployment so the scheduler can account for disk consumption, add a node disk usage alert well below the kubelet hard eviction threshold, and configure kubelet image garbage collection thresholds so unused image layers are reclaimed before disk pressure builds. None of these controls were in place in this incident.