Kubernetes root cause analysis guides
Root cause analysis guides for Kubernetes production failures. Each covers one failure mode: what the pod state means, the commands that confirm it, and the configuration change that stops it recurring.
sre@prod-bastion ~ $ kubectl get pods -n prodNAME READY STATUS AGElanding-page 0/1 Evicted 4mcheckout-api 0/1 CrashLoopBackOff 12mkyc-worker 0/1 OOMKilled 8mnotify-svc 0/1 CreateContainerConfigError 2m sre@prod-bastion ~ $ kubectl describe pod landing-page
In this series
(05 Guides)How to debug Kafka consumer lag in Kubernetes?
Kafka consumer lag climbing while every consumer looks healthy usually means partition skew, a rebalance storm from pod churn or HPA scaling, or max.poll.interval.ms ejecting a live consumer. This guide covers each cause and why adding replicas can make lag worse.
How to fix a Kubernetes CrashLoopBackOff?
CrashLoopBackOff means a container keeps starting and dying, so Kubernetes waits longer between restarts. This guide covers exit codes 1, 137, and 143, reading the dead container with kubectl logs --previous, and telling a failed dependency apart from a code fault.
How to fix a Kubernetes OOMKilled container (exit code 137)?
An OOMKilled container with exit code 137 means it exceeded its own memory limit and the kernel stopped it. This guide covers what causes it, how to tell it apart from node pressure, how to confirm it, and how to fix it for good.
How to fix a Kubernetes pod evicted for ephemeral storage?
Why a pod is evicted for ephemeral-storage and how to stop it recurring. Covers the nodefs.available<10% threshold, the DiskPressure taint that leaves replacements Pending, finding the writer with du and the kubelet Summary API, and the ephemeral-storage requests, emptyDir sizeLimit and LimitRange that fix it.
How to fix Kubernetes CreateContainerConfigError?
CreateContainerConfigError means the image pulled fine but Kubernetes could not assemble the container's config, usually a missing Secret or an absent ConfigMap key. This guide traces the reference, explains how envFrom fails differently from valueFrom, and covers the namespace mismatch that hides the cause.
Each guide ends at a real incident. The matching Kubernetes investigations show the same failures worked end to end, hypotheses and all.
Other stacks
- AWSSoon
- DatabasesSoon
- Linux VMsSoon
See Sherlocks AI in action
Watch an AI SRE work a real incident from alert to root cause, on your stack, in 30 minutes.