Kubernetes failures rarely announce their cause. A pod in CrashLoopBackOff, a container stuck in CreateContainerConfigError, and a service returning 503 all look like application failures from the outside, and all three routinely turn out to be something else: a missing Secret, a scaling policy, a node running out of ephemeral storage. The state name describes what the kubelet is doing, not what went wrong. Four of the investigations below open on CrashLoopBackOff and end in causes with nothing in common: an AMQP broker that was unreachable, a mutex assertion in application code, contention for cluster resources, and a missing JVM argument for a connector. Two others start inside the cluster and finish outside it, at an Aurora load surge and at an ElastiCache connection burst driven by HPA pod churn, because the cluster is frequently where an external event first becomes visible. The control plane reports symptoms accurately and says almost nothing about causes. These are real production incidents, each worked from the opening alert to a confirmed root cause, including the hypotheses that turned out to be wrong.
Pod evicted by kubelet after node ran out of ephemeral-storage (510 Mi available vs 2.14 GB threshold). DiskPressure taint blocked replacements; Karpenter provisioned new capacity. Missing ephemeral-storage requests allowed silent overcommit.
A manual show_manual_review_kyc request processed 22.36M KycStatusLog rows in-memory, breaching the 2.5 GB container limit. The pod was OOMKilled (Exit 137) and readiness probes failed. Node memory sat at 3.12%, so the breach was purely container-local, not node pressure.
HPA-driven pod churn on eaze-channels pushed CurrConnections from ~280 to 1,788 on Redis for one 5-minute bucket. Cache health stayed normal; transient connection fan-out from scaling, not a deploy or resource saturation.
Celery worker hit 91% of 1Gi memory limit as task backlog grew during a concurrent Aurora PostgreSQL load surge. Pod was evicted for Underutilized (not OOMKilled); surviving replica still at 87%. Thin headroom, not a crash.
Celery consumer entered CrashLoopBackOff during startup after failing to establish AMQP/Kombu broker connection. Pod was replaced and deployment self-healed. eaze also affected but root cause not preserved in logs.
Prometheus PVC filled up due to TSDB block growth from increased metric cardinality under a 65-day retention window. Pod healthy: organic storage growth, not WAL bloat.
Container stuck in CreateContainerConfigError for 22+ minutes because a required Kubernetes Secret was missing from the namespace. Image pulls fine: config assembly blocked.
Deployment scaled replicas to 0, leaving the service with no healthy endpoints. Root cause identified in under 5 minutes.
Rollout churn combined with CPU throttling and HPA hard-cap prevented consumer recovery, causing 35.9K message lag.
Fatal mutex assertion failure in runtime triggered by deployment revision 1848, causing repeated pod crashes with 7 restarts.
Trino coordinator and worker pods entered CrashLoopBackOff during cluster-wide resource contention (insufficient memory/cpu). The Image Pull portion of the Grafana alert was a false positive: pulls of trinodb/trino:475 succeeded. A manual rollout restart replaced the failing ReplicaSets and recovered service.
Trino pods crash on startup (Exit Code 100) because the BigQuery connector requires --add-opens=java.base/java.nio=ALL-UNNAMED, which is missing from JVM config. Image Pull alert was a false positive.
Most Kubernetes incidents separate quickly on three questions. First, did the container ever run? A Waiting state with a zero restart count means the process never started, which points at image pulls, config assembly, or admission, not at application code. In the CreateContainerConfigError case here the image pulled cleanly in 127 ms and the failure was config assembly against a Secret that did not exist. A Terminated state with restarts points at the process itself, and the exit code narrows it further: 137 with Reason=OOMKilled is the kernel killing the process, which is a different failure from a runtime reporting that it ran out of heap. Second, was the pressure container-local or node-wide? In the OOMKilled query investigation, node memory sat at 3.12 percent while the container breached its 2.5 GB cgroup limit, so the fix belonged in the workload rather than the cluster. Third, did anything change? Deployment revision history and restart timestamps usually bracket the incident tightly enough to separate a code change from an infrastructure event. Not every outage has a crash behind it: one 503 here had empty endpoints and no pods backing the service at all, a deployment scaled to zero replicas. Working those questions in order rules out most of the hypothesis space before any deep inspection begins.