Alert info
- Severity
- Critical
- Detected by
- Kubernetes
- Alert
- Pod in CrashLoopBackOff, 222 restarts, no error output in the container log
- Time
- August 2026 (exact timestamp not preserved in this write-up)
- Service
- checkout-api (prod)
The pod restarted 222 times. Each container log held two initialisation lines and then stopped. No stack trace, no exception, and no error text anywhere in the stream. Every signal the cluster produced described the workload, and the workload had not been changed.
A revoked security group rule cut the cluster off from Postgres. The application treated the database as a hard startup dependency, exited before binding its port, and never logged the failure, so 222 restarts pointed at the application while the change sat in the network layer.
- Restarts
- 222
- Errors logged
- 0
- Time to causal chain
- 16 min
Entities identified
Infrastructure
kubelet
restart policy
Restarts the exited container and backs off, which is why the failure presented as a climbing restart count rather than a stopped workload.
security group
revoked rule
Ingress rule permitting the cluster to reach the database port was revoked.
Service
checkout-api
prod
Application pod in CrashLoopBackOff with 222 restarts.
startup sequence
config then DB then bind
Loads configuration, acquires a database connection, then binds its listening port. The database step is a hard dependency.
listening port
never bound
The process exited before reaching the bind, so no port was ever opened for a probe to answer.
Data
Postgres
managed instance
The database the application requires before it will start.
Gateway
infrastructure change
outside the deploy pipeline
Security group edits are not recorded in the deploy timeline, so the usual
what-shipped-recentlycorrelation returns nothing.
How they relate
The same relationships as text
- kubelet restarts checkout-api. 222 restart cycles, each producing an identical truncated log.
- checkout-api runs startup sequence
- startup sequence requires Postgres. Hard startup dependency. The process will not proceed without a connection.
- startup sequence never reached listening port. Output stops between configuration load and port bind.
- security group guards path to Postgres. The revoked rule is what closed the route between the cluster and the database port.
- infrastructure change revoked rule security group
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Where in startup does the process stop?Confirmed
Read the container log for the crashed pod and establish which startup stage produced the last line before the process exited.
Output stops after two initialisation lines. That bounds the failure to the window between configuration load and port bind, and rules out everything the application does after it starts serving. The absence is the measurement here: a process that dies inside that window is telling you which dependency it was acquiring.
Two initialisation lines, then no further output. Identical across restarts.
Failure sits between config load and port bind. Nothing after the bind executed.
222 restarts, climbing, with no change in log shape between them.
A persistent condition rather than a transient one. Backoff was retrying a fault that would never clear on its own.
Container never bound its listening portConfirmed
Determine whether the process reached its listen call, and whether probe failures carried any additional detail.
No port was ever opened, so the probe failure reported only that nothing was answering. That is the same fact the restart count already carried, which is why the probe added no information and why both signals pointed at the workload.
Probe failing against a port that was never opened.
Accurate and useless. It describes the symptom, not the cause.
Application regression: code, image, or configuration changeNeeds access
Attempt to correlate the onset against deployment revision history, image digests, and recent configuration changes.
Not available in this write-up. The deploy timeline is the first thing consulted during an incident of this shape, and the reason this class of failure misleads is that a network change leaves no record in it. Confirming that nothing shipped requires the revision history and the image provenance.
Revision history and image digests need access to confirm.
Absence of a deploy in the timeline is weak evidence on its own, because the change that mattered was never going to appear there.
Database dependency unreachable at startupConfirmed
Test reachability to the database from inside the cluster, from a pod in the same namespace as the failing workload, rather than from an operator laptop with its own route.
The path was closed from where the application actually runs. Testing from anywhere else would have succeeded and sent the investigation back to the application, which is the trap this check exists to avoid.
Connection to the database port did not succeed from inside the cluster.
The dependency is unreachable at the network layer, not misconfigured in the application.
Postgres itself down, saturated, or refusing connectionsNeeds access
Attempt to check database availability, connection counts, and server logs across the incident window.
Outside the captured telemetry for this write-up. A healthy database behind a closed path and an unhealthy database present identically to a client that only sees a timeout, so separating them requires server-side visibility.
Server-side availability and connection metrics need access.
Not required to establish the cause here, but it is the branch that would otherwise stay open.
Network path closed by a revoked security group ruleRoot cause
Inspect infrastructure changes affecting the path between the cluster and the database, then confirm against the reachability result.
An ingress rule permitting the cluster to reach the database port had been revoked. That single change produced every symptom above: the connection attempt hung rather than being refused, the startup sequence never completed, the process exited before binding, and the kubelet restarted it 222 times. The change was made deliberately and carefully, outside the deploy pipeline, which is why nothing in the application timeline reflected it.
Ingress rule allowing cluster to database access had been revoked.
A connectivity change is invisible from where it is made. The person closing an overly broad rule had no view of which workloads depended on it.
Final RCA
Network path closed by a revoked security group rule
An ingress rule permitting the cluster to reach the database port had been revoked. That single change produced every symptom above: the connection attempt hung rather than being refused, the startup sequence never completed, the process exited before binding, and the kubelet restarted it 222 times. The change was made deliberately and carefully, outside the deploy pipeline, which is why nothing in the application timeline reflected it.
Remediation
# Restore the ingress rule permitting the cluster to reach the database port
# Then confirm from inside the cluster, not from a laptop:
kubectl run netcheck --rm -it --restart=Never --image=busybox -n <namespace> -- nc -zv <db-host> 5432- Log the dependency before acquiring it. A line naming the host and port the process is about to reach turns the silence into a sentence.
- Emit a structured error on the database failure path, carrying host, port, and timeout, rather than exiting quietly.
- Name the startup stages in the log so the last line printed identifies the phase the process reached.
- Treat an empty deploy timeline during an incident as information rather than a dead end. Something changed; it changed somewhere the pipeline does not record.
A hard startup dependency is a reasonable design, and it becomes a trap only when it is implicit. Say what you are about to require, say what you failed to reach, and say it on the path that exits. Separately, treat security group, network policy, and IAM edits as deployments: they change production, they are made outside the pipeline, and an incident timeline that cannot see them will point at the wrong layer every time.
Frequently asked questions
Why did the pod restart 222 times instead of failing permanently?
Kubernetes restarts a container that exits, and repeated failures move the pod into CrashLoopBackOff, which spaces the attempts out but never stops trying. That is correct behaviour for a transient fault, and it is what makes a persistent one easy to sit with: the cluster looks busy working the problem while the underlying condition never clears.
Why was there no error in the logs?
The failure happened during startup, before the code that produces useful error output had anything to say. The connection attempt hung and timed out rather than being refused, and the handler for that path exited the process instead of logging a structured error naming what it had been trying to reach. The two lines present came from earlier in the sequence.
Would a liveness or readiness probe have caught this sooner?
It caught the symptom, which was already visible. Probes report that a port is not answering, not why. Here the probe failed for the same reason everything else did, because the process exited before it ever bound its port. The probe result and the restart count were both accurate and both pointed at the application.
How do you separate an application fault from a network change?
Test reachability from where the application actually runs, using a pod in the same namespace rather than an operator laptop that may have its own route. If the path is closed in cluster but open elsewhere, the fault is in the network layer. Then inspect infrastructure changes, which are timestamped even when they never touch the deploy pipeline.