Kubernetes root cause analysis guides

Kubernetes is unusually good at telling you precisely what it did. It is not trying to tell you why, and reading the status as if it were a diagnosis is where most of the time goes.

That is the difference from a plain VM, where nothing announces itself and you go looking. Here a short, confident string appears in kubectl get pods, it is completely correct, and it describes the control plane's reaction rather than the thing it reacted to. The whole skill is knowing which layer to look at next.

Why does the pod status never tell you the cause?#

Because the status field reports a lifecycle transition, and lifecycle transitions are reactions.

CrashLoopBackOff is the clearest example. It does not mean the container failed for a particular reason. It means the container has already failed several times and the kubelet is now spacing the restarts out, doubling the delay each round. By the time the status appears, the container that holds the evidence is gone, replaced by one that has not failed yet.

# The status is on the current container. The reason is in the one# that already died, which is what --previous reads.kubectl logs checkout-api -n prod --previous --tail=50 # The exit code narrows it before you read a single log line.kubectl describe pod checkout-api -n prod | grep -A5 'Last State' 

Last State:     Terminated  Reason:       Error  Exit Code:    1  Started:      Mon, 30 Mar 2026 11:18:42 +0530  Finished:     Mon, 30 Mar 2026 11:18:44 +0530 

Exit codes are the fastest triage in Kubernetes. 1 is the application deciding to stop, so the cause is in its logs. 137 is SIGKILL, which almost always means the kernel enforced a memory limit. 143 is SIGTERM, meaning something asked the container to stop and it did. Three numbers separate an application bug from a memory ceiling from an orchestration decision, before you have read anything.

The same pattern holds across the other states. CreateContainerConfigError means the image pulled cleanly and Kubernetes then could not assemble the container's configuration, which is a reference to an object that is not there rather than anything wrong with your code. Evicted means the kubelet reclaimed a node resource. In none of these cases is the failing thing the thing named in the status.

Which object actually hit the ceiling?#

This is the question that decides whether you are debugging a container, a pod, or a node, and getting it wrong sends the whole investigation into the wrong place.

Kubernetes enforces limits at several levels at once, and two of them produce nearly identical symptoms. A container exceeding its own memory limit is killed by the kernel through its cgroup, with the rest of the node perfectly healthy. A node running low on memory or disk causes the kubelet to start evicting pods, which may include pods that were well inside their own limits and simply had the misfortune to be there.

# Container-local: did this container breach its own limit?kubectl describe pod kyc-worker -n prod | grep -E 'Reason|Exit Code|memory' # Node-level: is the node itself under pressure? DiskPressure or# MemoryPressure being True changes the answer completely.kubectl describe node ip-10-0-3-14 | grep -A6 Conditions 

# container-localLast State:  Terminated  Reason:    OOMKilled  Exit Code: 137  Limits:    memory: 2500Mi # node-levelType             StatusMemoryPressure   FalseDiskPressure     True     <- the node, not the pod 

OOMKilled with the node reporting no pressure is a limit that is too low or a workload that grew, and it is fixed in the container's resources or in the query that loaded too much. A pod Evicted with DiskPressure true is a node story: the kubelet crossed nodefs.available<10%, tainted the node, and started reclaiming. The evicted pod is often not the pod that filled the disk, and replacements will sit Pending until the taint clears.

Read the direction of causation before you change anything. Raising a memory limit on a pod that was evicted for node disk changes nothing at all.

How do you triage a Kubernetes incident?#

Four moves, in order, and the first two eliminate most of the search space.

Read the exit code and the previous container before anything else. The status names the loop; the dead container holds the reason. kubectl logs --previous plus the exit code from Last State separates an application fault from a kill signal in one command, and it costs nothing.

Establish whether the ceiling was the container's or the node's. Compare the container's Reason against the node's Conditions. A cgroup kill and a kubelet eviction look similar in a dashboard and have opposite fixes. This is the check that stops you tuning the wrong object.

# When several pods are unhappy at once, the node is the better# first question. Events are ordered, so the sequence tells you# whether pods failed and then the node degraded, or the reverse.kubectl get events -n prod --sort-by=.lastTimestamp | tail -20 # Which pods share a node with the failing one? Co-location is the# fastest way to distinguish a workload problem from a host problem.kubectl get pods -n prod -o wide --field-selector spec.nodeName=ip-10-0-3-14 

When the object is configuration, trace the reference rather than reading the manifest. A CreateContainerConfigError is a pointer to something absent, and the absence is what you have to prove. Check the object exists, then check the specific key exists inside it, then check you are looking in the right namespace. envFrom and valueFrom fail differently, which is why the error text alone is not enough.

# The object, then the key inside it. A Secret that exists with the# wrong key set fails exactly like a Secret that does not exist.kubectl get secret analytics-config -n prodkubectl get secret analytics-config -n prod -o jsonpath='{.data}' | tr ',' '\n' 

For throughput problems, check distribution before capacity. Consumer lag climbing while every consumer reports healthy is rarely slow code. It is usually skew, where a few partitions carry most of the traffic, or a rebalance storm caused by pod churn or an HPA cycling replicas. Both get worse when you add replicas, because every new member triggers another rebalance and the group spends its time re-joining rather than consuming.

# Per-partition lag. A group total hides the shape, and the shape# is the diagnosis: even lag is capacity, concentrated lag is skew.kafka-consumer-groups.sh --bootstrap-server kafka:9092 \  --describe --group kyc-consumers # Rebalances leave a trail in the consumer's own logs.kubectl logs -n prod -l app=kyc-consumer --tail=200 | grep -i 'rebalanc\|revoked\|joining' 

That last one generalises. Across Kubernetes incidents, the instinct to add capacity is right roughly as often as it is wrong, and the way to tell is whether the resource is evenly consumed. Evenly saturated means capacity. Unevenly saturated means distribution, and more replicas will not fix distribution.

The Kubernetes troubleshooting guides#

Each guide covers one failure mode end to end: what the state means, the commands that confirm it, and the configuration change that stops it recurring.

How to fix a Kubernetes CrashLoopBackOff. Reading the dead container, what exit codes 1, 137 and 143 each imply, and telling a failed dependency apart from a code fault.

How to fix a Kubernetes OOMKilled container. Why exit code 137 is nearly always the container's own cgroup limit rather than node memory, and how to confirm it before changing a limit.

How to fix a Kubernetes pod evicted for ephemeral storage. The nodefs.available<10% threshold, the DiskPressure taint that leaves replacements Pending, and finding the actual writer on the node.

How to fix Kubernetes CreateContainerConfigError. Tracing the reference to the missing Secret or ConfigMap key, how envFrom fails differently from valueFrom, and the namespace mismatch that hides the cause.

How to debug Kafka consumer lag in Kubernetes. Partition skew, rebalance storms from pod churn or HPA scaling, max.poll.interval.ms ejecting a live consumer, and why adding replicas can make lag worse.

You can see these worked end to end, including the hypotheses that were ruled out, on the Kubernetes examples page. When the ceiling turns out to be a managed service rather than the cluster, the AWS guides cover the per-dimension limits that produce the same symptoms, and when a container is killed for memory on a plain host instead of in a cgroup, JVM memory leaks and Linux OOM kills covers how that failure differs. For the mechanics themselves, the pod lifecycle documentation and the node-pressure eviction documentation are the primary references.

Frequently asked questions#

What does CrashLoopBackOff actually mean?

That a container has failed repeatedly and the kubelet is now spacing out its restarts. It is a restart policy, not a cause. The reason is in the previous container, which kubectl logs --previous reads.

What is the difference between OOMKilled and a pod being evicted?

OOMKilled is one container exceeding its own memory limit, enforced by its cgroup, on a node that may be perfectly healthy. Eviction is the kubelet reclaiming a node-level resource, and the pod it removes is often not the one that consumed it.

What does exit code 137 mean in Kubernetes?

The container received SIGKILL, which in practice almost always means it breached its memory limit. Compare it against exit code 1, an application error, and 143, a SIGTERM the container honoured.

Why does CreateContainerConfigError happen when the image pulled fine?

Because the failure is after the pull. Kubernetes could not assemble the container's configuration, usually because a referenced Secret or ConfigMap key does not exist, or exists in a different namespace.

Why is my consumer lag climbing when every consumer looks healthy?

Usually partition skew or a rebalance storm rather than slow processing. Check per-partition lag rather than the group total, and check whether pod churn or an HPA is triggering repeated rebalances.

Will adding replicas fix consumer lag?

Often it makes it worse. Every new group member triggers another rebalance, and replicas beyond the partition count sit idle. Add replicas only once you have confirmed lag is evenly distributed.