Alert info
- Severity
- Warning
- Detected by
- Grafana
- Alert
- Image Pull / CrashLoopBackOff on content-tagger-consumer and eaze pods
- Time
- 2026-04-17 04:19:28 UTC
- Service
- content-tagger-consumer (prod)
content-tagger-consumer and eaze fired CrashLoopBackOff at 04:19:28Z. By investigation time both alerted pods were NotFound and both deployments were back to full availability, so this was a real but transient startup crash.
Celery consumer entered CrashLoopBackOff during startup after failing to establish AMQP/Kombu broker connection. Pod was replaced and deployment self-healed. eaze also affected but root cause not preserved in logs.
- Workloads Hit
- 2
- Crash Reason
- AMQP failure
- Recovery
- Self-healed
Entities identified
Infrastructure
consumer_start.sh
startup command
Startup command .
/server_launcher/consumer_start.shfor the consumer process.Kubernetes
prod control plane
Restarted the failing pods on CrashLoopBackOff and replaced them. Both deployments recovered to full availability. Event history did not retain the original BackOff events; Prometheus crash-loop series returned no data.
Service
content-tagger-consumer
Celery worker
Celery consumer, revision 1009, image api-backend:2026-04-14-21-10-112. Entered CrashLoopBackOff at startup. Now 1/1 Running, 0 restarts (self-healed).
eaze
web workload
Revision 403, image eaze:2026-04-15-17-41-329, restartedAt 04:19:54Z. Also reported in CrashLoopBackOff. Logs showed only Gunicorn startup + JWT warnings; no fatal stacktrace preserved. Now 5/5 Running.
Data
ELK (eks-prod-logs)
log store
Log store searched 04:05Z-04:25Z. Held repeated AMQP/Kombu connection-failure tracebacks for the consumer; only Gunicorn/JWT lines for eaze.
External
RabbitMQ / AMQP
message broker
Message broker over AMQP (port 5672). Responsible for message queuing.
Identity
broker secret
BROKER_URL
content-tagger-consumer-brokerSecret holding BROKER_URL (AMQP credentials + endpoint). Not audited during the investigation; rotation near 04:18Z cannot be excluded.
Gateway
lokalapp-prod
caller
Upstream caller that enqueues tagging work onto the consumer. No downstream of its own; async work may have dropped during the restart window.
How they relate
The same relationships as text
- lokalapp-prod enqueues work content-tagger-consumer
- content-tagger-consumer launched by consumer_start.sh
- consumer_start.sh connects (AMQP) RabbitMQ / AMQP. Kombu opens the AMQP transport to the broker at startup.
- RabbitMQ / AMQP ConnectionForced consumer_start.sh. Broker responds to AMQP transport connection.
- broker secret supplies BROKER_URL consumer_start.sh
- content-tagger-consumer logs to ELK (eks-prod-logs)
- eaze logs to ELK (eks-prod-logs)
- Kubernetes restarts / replaces content-tagger-consumer
- Kubernetes restarts / replaces eaze
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Image pull / new image regression crashed the consumer at startupRuled out
The alert names Image Pull. Diff image tags and dates against the alert time for both workloads.
content-tagger-consumer image is dated 2026-04-14, eaze 2026-04-15; neither changed at alert time. The crash is an AMQP traceback at connect, not an import or pull error.
Both images predate the alert by days; no pull failure recorded.
workload revision image dated
content-tagger-consumer 1009 api-backend:2026-04-14-21-10-112 2026-04-14
eaze 403 eaze:2026-04-15-17-41-329 2026-04-15
No image change at the 04:19Z alert. The Image Pull label is a misnomer; the real signal is the crash loop.
Node pressure / eviction (OOM or ephemeral storage) forced the restartRuled out
Scan kubectl events for OOMKilled, Evicted, or node-pressure on the affected pods.
No OOMKilled, Evicted, or node-pressure events. The crash reason is a Celery-level connection traceback, not a kernel OOM kill or kubelet eviction.
Only unrelated HPA/ingress sync noise returned for the window.
$ kubectl get events -n prod --sort-by=.lastTimestamp | tail -n 50
# HPA scaling events for other deployments
# Ingress sync noise
# No OOMKilled / Evicted / node-pressure for consumer or eaze
No eviction or memory-pressure signal. The original BackOff events rolled off after pod replacement.
Manual operator restart, not an infrastructure fault, drove the loopRuled out
Check whether a manual rollout restart explains the crash itself by inspecting the restartedAt annotations for both deployments in the window.
A restart around 04:18Z-04:19Z may have occurred, but it does not explain the AMQP traceback: even a clean restart hit the same broker-refused failure. The restart only surfaced the connectivity issue. Ruled out as the cause.
Both workloads restarted within seconds of the alert.
content-tagger-consumer restartedAt 2026-04-17T04:18:18Z
eaze restartedAt 2026-04-17T04:19:54Z
alert fired 2026-04-17T04:19:28Z
Restart timing overlaps the alert, but the failure mode is broker connect, not the restart itself.
Consumer process exited at startup on an AMQP/Kombu broker connection failureConfirmed
Search ELK (eks-prod-logs, 04:05Z-04:25Z) for content-tagger-consumer startup tracebacks and identify where the process dies.
ELK holds repeated Celery/Kombu AMQP transport connection-failure tracebacks ending in amqp.exceptions.ConnectionForced. The crash is at the broker connect path during startup, not mid-run. Confirmed; trace why the process exits rather than retries.
content-tagger-consumer tracebacks fail in the Kombu connect path during startup.
ELK - content-tagger-consumer (04:05Z - 04:25Z)
Pattern Count Level
AMQP / kombu transport connection failure repeated ERROR
Traceback (most recent call last):
File ".../celery/worker/consumer/consumer.py", line ...
self.connection = self.connect()
File ".../kombu/connection.py", line ...
return self.transport.establish_connection()
amqp.exceptions.ConnectionForced: broker connection refused
Failure is at connect, in celery/worker/consumer/consumer.py, not a mid-run disconnect. The broker refused the startup connection.
consumer_start.sh exits on first broker refusal with no retry, so K8s loopsRoot cause
Inspect the startup command and the restart/recovery pattern: does a single refused connection terminate the process?
./server_launcher/consumer_start.sh has no startup retry/backoff: one refused AMQP connection kills the process, Kubernetes restarts it, and the cycle becomes CrashLoopBackOff. Both alerted pods were then replaced and the deployments self-healed (1/1 and 5/5).
Startup script exits on broker failure; deployments recovered after pod replacement.
command: ./server_launcher/consumer_start.sh # exits on broker connect failure
$ kubectl get deploy -n prod -l 'app in (content-tagger-consumer, eaze)'
NAME READY UP-TO-DATE AVAILABLE
content-tagger-consumer 1/1 1 1
eaze 5/5 5 5
No retry means one refusal triggers CrashLoopBackOff; once the broker recovered, replacement pods came up clean.
Restart count and crash-loop duration could not be quantified.
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff",
pod=~"content-tagger-consumer.*|eaze.*", namespace="prod"}
-> no data
kube_pod_container_status_restarts_total{...} -> no data
Time-series gap; the crash loop is evidenced by logs and rollout annotations, not Prometheus.
Why the broker refused the connection (restart, auth, DNS, or network)Needs access
Attempt to query RabbitMQ broker-side connection/auth logs and audit the BROKER_URL secret rotation near 04:18Z to determine the trigger.
The precise broker-side cause was not preserved in our telemetry. A transient broker restart is most likely given the rapid self-heal, but credential rotation, DNS, or a network blip cannot be excluded. eaze shows the same timing but no fatal trace, so its attribution is also open.
No RabbitMQ-side telemetry; broker secret not audited; eaze logs held no fatal stacktrace.
ELK - eaze (04:05Z - 04:25Z)
Pattern Count Level
Gunicorn startup lines multiple INFO
JWT / Python runtime warnings present WARN
Fatal stacktrace / crash error 0 ERROR
The investigation stops at the AMQP boundary and labels the limit rather than guessing the broker-side trigger.
Network partition between the Kubernetes cluster and the AMQP brokerRuled out
Check VPC Flow Logs and NAT gateway metrics for packet drops or connection resets.
No broad network partition was observed. Other connections to the broker from different services succeeded during this window.
Liveness probe is overly aggressive and kills the pod during slow startupRuled out
Review the pod description for liveness probe failures during startup.
No liveness probe failures were recorded. The process itself exited with code 1.
No Unhealthy events for liveness probes.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 11m kubelet Started container
Warning BackOff 10m (x6 over 12m) kubelet Back-off restarting failed container
Pod is crashing independently of liveness checks.
Final RCA
consumer_start.sh exits on first broker refusal with no retry, so K8s loops
./server_launcher/consumer_start.sh has no startup retry/backoff: one refused AMQP connection kills the process, Kubernetes restarts it, and the cycle becomes CrashLoopBackOff. Both alerted pods were then replaced and the deployments self-healed (1/1 and 5/5).
Remediation
# Check RabbitMQ/broker health and recent restarts
kubectl get pods -n prod -l app=rabbitmq --sort-by=.metadata.creationTimestamp
# Verify AMQP credentials/endpoint in the consumer secret
kubectl get secret content-tagger-consumer-broker -n prod -o jsonpath='{.data.BROKER_URL}' | base64 -d
# Check if consumer_start.sh has retry/backoff logic on broker connect
kubectl exec -n prod deploy/content-tagger-consumer -- cat ./server_launcher/consumer_start.sh- Add startup retry/backoff logic to
consumer_start.sh so transient broker unavailability does not immediately crash the container and trigger CrashLoopBackOff. - Configure a Kubernetes startupProbe that checks broker connectivity, giving the container time to retry before Kubernetes marks it as failed.
- Add RabbitMQ/broker health monitoring (connection count, queue depth, node status) with alerts that fire before downstream consumers are affected.
- Audit recent credential/secret rotations and config changes that may have overlapped the 04:18Z restart window.
- Investigate the eaze CrashLoopBackOff separately: enable verbose startup logging or add structured error reporting to capture fatal exits.
The startup script exits on the first refused AMQP connection with no retry or backoff, so any transient broker hiccup, whether a restart, a credential rotation, or a brief network blip, turns into a full CrashLoopBackOff instead of a quiet retry. Adding retry/backoff to consumer_start.sh and a startupProbe that checks broker connectivity before Kubernetes judges the container failed removes that single point of fragility. Because the eaze CrashLoopBackOff in the same window was never attributed and the Prometheus crash-loop series returned no data, the second control worth adding is broker-side health monitoring and verbose startup logging, so the next occurrence leaves evidence instead of self-healing into a gap.
Frequently asked questions
What causes a Celery worker to enter CrashLoopBackOff on AMQP connection failure?
In this incident, ELK logs showed the Celery consumer's Kombu transport failing to establish an AMQP connection at startup, ending in amqp.exceptions.ConnectionForced. The startup script, consumer_start.sh, has no retry or backoff logic, so a single refused connection terminates the process. Kubernetes then restarts the container, and because the broker refusal repeats immediately, the pod cycles into CrashLoopBackOff.
Why couldn't the exact cause of the AMQP broker refusal be confirmed?
Because the investigation stopped at the boundary of available telemetry. There was no RabbitMQ-side connection or auth log to inspect, and the BROKER_URL secret was not audited for a rotation near the 04:18Z restart window. A transient broker restart is the most likely explanation given how quickly both workloads self-healed, but credential rotation, DNS, or a brief network issue could not be ruled out from the data pulled.
How do you confirm a CrashLoopBackOff was a real crash and not just an image pull problem?
Check the image tags and dates against the alert time, and read the actual traceback rather than trusting the alert label. In this incident the alert was tagged Image Pull, but both workloads' images predated the alert by one to three days with no pull failure recorded, while ELK logs showed a Celery/Kombu AMQP connection traceback at startup. The alert label was a misnomer; the real failure was a broker connection refusal.
Does a pod recovering on its own mean the CrashLoopBackOff cause was fixed?
Not necessarily. In this incident both content-tagger-consumer and eaze were fully healthy by the time the investigation began (1/1 and 5/5 available), Kubernetes had already replaced the failing pods, and the original BackOff events had rolled off so Prometheus crash-loop metrics returned no data. Self-healing through pod replacement is not the same as fixing the missing retry logic that let one refused connection cascade into a crash loop in the first place.