CrashLoopBackOff with no error output: a revoked security group rule

Alert info

Severity
Critical
Detected by
Kubernetes
Alert
Pod in CrashLoopBackOff, 222 restarts, no error output in the container log
Time
August 2026 (exact timestamp not preserved in this write-up)
Service
checkout-api (prod)

The pod restarted 222 times. Each container log held two initialisation lines and then stopped. No stack trace, no exception, and no error text anywhere in the stream. Every signal the cluster produced described the workload, and the workload had not been changed.

A revoked security group rule cut the cluster off from Postgres. The application treated the database as a hard startup dependency, exited before binding its port, and never logged the failure, so 222 restarts pointed at the application while the change sat in the network layer.

Restarts
222
Errors logged
0
Time to causal chain
16 min

Entities identified

Infrastructure

  • kubelet

    restart policy

    Restarts the exited container and backs off, which is why the failure presented as a climbing restart count rather than a stopped workload.

  • security group

    revoked rule

    Ingress rule permitting the cluster to reach the database port was revoked.

Service

  • checkout-api

    prod

    Application pod in CrashLoopBackOff with 222 restarts.

  • startup sequence

    config then DB then bind

    Loads configuration, acquires a database connection, then binds its listening port. The database step is a hard dependency.

  • listening port

    never bound

    The process exited before reaching the bind, so no port was ever opened for a probe to answer.

Data

  • Postgres

    managed instance

    The database the application requires before it will start.

Gateway

  • infrastructure change

    outside the deploy pipeline

    Security group edits are not recorded in the deploy timeline, so the usual what-shipped-recently correlation returns nothing.

How they relate

restartsrunsrequiresnever reachedrevoked rulekubeletrestart policyinfrastructure changeoutside the deploy pipeli…checkout-apiprodsecurity grouprevoked rulestartup sequenceconfig then DB then bindlistening portnever boundPostgresmanaged instance
The same relationships as text
  • kubelet restarts checkout-api. 222 restart cycles, each producing an identical truncated log.
  • checkout-api runs startup sequence
  • startup sequence requires Postgres. Hard startup dependency. The process will not proceed without a connection.
  • startup sequence never reached listening port. Output stops between configuration load and port bind.
  • security group guards path to Postgres. The revoked rule is what closed the route between the cluster and the database port.
  • infrastructure change revoked rule security group

Hypotheses tree

Ruled-out branches are collapsed. Open any one to read the check that eliminated it.

Where in startup does the process stop?Confirmed

Read the container log for the crashed pod and establish which startup stage produced the last line before the process exited.

Output stops after two initialisation lines. That bounds the failure to the window between configuration load and port bind, and rules out everything the application does after it starts serving. The absence is the measurement here: a process that dies inside that window is telling you which dependency it was acquiring.

Container log, from kubectl logs --previous

Two initialisation lines, then no further output. Identical across restarts.

Failure sits between config load and port bind. Nothing after the bind executed.

Restart count, from Pod status

222 restarts, climbing, with no change in log shape between them.

A persistent condition rather than a transient one. Backoff was retrying a fault that would never clear on its own.

Container never bound its listening portConfirmed

Determine whether the process reached its listen call, and whether probe failures carried any additional detail.

No port was ever opened, so the probe failure reported only that nothing was answering. That is the same fact the restart count already carried, which is why the probe added no information and why both signals pointed at the workload.

Probe result, from Pod status

Probe failing against a port that was never opened.

Accurate and useless. It describes the symptom, not the cause.

Application regression: code, image, or configuration changeNeeds access

Attempt to correlate the onset against deployment revision history, image digests, and recent configuration changes.

Not available in this write-up. The deploy timeline is the first thing consulted during an incident of this shape, and the reason this class of failure misleads is that a network change leaves no record in it. Confirming that nothing shipped requires the revision history and the image provenance.

Telemetry boundary, from Deploy history

Revision history and image digests need access to confirm.

Absence of a deploy in the timeline is weak evidence on its own, because the change that mattered was never going to appear there.

Database dependency unreachable at startupConfirmed

Test reachability to the database from inside the cluster, from a pod in the same namespace as the failing workload, rather than from an operator laptop with its own route.

The path was closed from where the application actually runs. Testing from anywhere else would have succeeded and sent the investigation back to the application, which is the trap this check exists to avoid.

In-cluster reachability test, from Pod in the same namespace

Connection to the database port did not succeed from inside the cluster.

The dependency is unreachable at the network layer, not misconfigured in the application.

Postgres itself down, saturated, or refusing connectionsNeeds access

Attempt to check database availability, connection counts, and server logs across the incident window.

Outside the captured telemetry for this write-up. A healthy database behind a closed path and an unhealthy database present identically to a client that only sees a timeout, so separating them requires server-side visibility.

Telemetry boundary, from Database server

Server-side availability and connection metrics need access.

Not required to establish the cause here, but it is the branch that would otherwise stay open.

Network path closed by a revoked security group ruleRoot cause

Inspect infrastructure changes affecting the path between the cluster and the database, then confirm against the reachability result.

An ingress rule permitting the cluster to reach the database port had been revoked. That single change produced every symptom above: the connection attempt hung rather than being refused, the startup sequence never completed, the process exited before binding, and the kubelet restarted it 222 times. The change was made deliberately and carefully, outside the deploy pipeline, which is why nothing in the application timeline reflected it.

Infrastructure change timeline, from Security group configuration

Ingress rule allowing cluster to database access had been revoked.

A connectivity change is invisible from where it is made. The person closing an overly broad rule had no view of which workloads depended on it.

Final RCA

Network path closed by a revoked security group rule

An ingress rule permitting the cluster to reach the database port had been revoked. That single change produced every symptom above: the connection attempt hung rather than being refused, the startup sequence never completed, the process exited before binding, and the kubelet restarted it 222 times. The change was made deliberately and carefully, outside the deploy pipeline, which is why nothing in the application timeline reflected it.

Remediation

# Restore the ingress rule permitting the cluster to reach the database port
# Then confirm from inside the cluster, not from a laptop:
kubectl run netcheck --rm -it --restart=Never --image=busybox -n <namespace> -- nc -zv <db-host> 5432
  • Log the dependency before acquiring it. A line naming the host and port the process is about to reach turns the silence into a sentence.
  • Emit a structured error on the database failure path, carrying host, port, and timeout, rather than exiting quietly.
  • Name the startup stages in the log so the last line printed identifies the phase the process reached.
  • Treat an empty deploy timeline during an incident as information rather than a dead end. Something changed; it changed somewhere the pipeline does not record.

A hard startup dependency is a reasonable design, and it becomes a trap only when it is implicit. Say what you are about to require, say what you failed to reach, and say it on the path that exits. Separately, treat security group, network policy, and IAM edits as deployments: they change production, they are made outside the pipeline, and an incident timeline that cannot see them will point at the wrong layer every time.

Frequently asked questions

Why did the pod restart 222 times instead of failing permanently?

Kubernetes restarts a container that exits, and repeated failures move the pod into CrashLoopBackOff, which spaces the attempts out but never stops trying. That is correct behaviour for a transient fault, and it is what makes a persistent one easy to sit with: the cluster looks busy working the problem while the underlying condition never clears.

Why was there no error in the logs?

The failure happened during startup, before the code that produces useful error output had anything to say. The connection attempt hung and timed out rather than being refused, and the handler for that path exited the process instead of logging a structured error naming what it had been trying to reach. The two lines present came from earlier in the sequence.

Would a liveness or readiness probe have caught this sooner?

It caught the symptom, which was already visible. Probes report that a port is not answering, not why. Here the probe failed for the same reason everything else did, because the process exited before it ever bound its port. The probe result and the restart count were both accurate and both pointed at the application.

How do you separate an application fault from a network change?

Test reachability from where the application actually runs, using a pod in the same namespace rather than an operator laptop that may have its own route. If the path is closed in cluster but open elsewhere, the fault is in the network layer. Then inspect infrastructure changes, which are timestamped even when they never touch the deploy pipeline.

Related investigations

We use cookies

Only the strictly necessary ones are set when you arrive. Analytics and marketing cookies are set solely if you allow them, and you can change your mind at any time. See our Privacy Policy.