Alert info
- Severity
- Critical
- Detected by
- Kubernetes
- Alert
- Pod ad-service-767c6f45c-xgvkb entered CrashLoopBackOff
- Time
- 2026-02-26 15:10 IST
- Service
- ad-service (prod)
Pod ad-service-767c6f45c-xgvkb has restarted 7 times in 21m and is in CrashLoopBackOff. Investigate the cause of the crashes.
Fatal mutex assertion failure in runtime triggered by deployment revision 1848, causing repeated pod crashes with 7 restarts.
- Restarts
- 7
- Pod Age
- 21m
- Revision
- 1848
Entities identified
Infrastructure
ad-service deploy
prod
Kubernetes Deployment
deploy/ad-servicein namespace prod.revision 1847
v2.14.1 (stable)
ReplicaSet
ad-service-5c7e8b3a1, image ad-service:v2.14.1. Ran 1d with 0 restarts. Now scaled to 0.revision 1848
v2.15.0-rc1
ReplicaSet
ad-service-767c6f45c. Created 21m ago; desired 1, ready 0.kubelet probes
liveness / readiness
Liveness http-get :8000/healthz, readiness http-get :8000/readyz.
Service
ad-service pod
767c6f45c-xgvkb
Pod
ad-service-767c6f45c-xgvkb. READY 0/1, STATUS CrashLoopBackOff, 7 restarts, age 21m.ad-service runtime
Mutex::Fer
App runtime inside the container.
Data
config.yaml
/etc/ad-service
Config at
/etc/ad-service/config.yaml.
Gateway
CI/CD promotion
image tag
CI/CD promotion path.
How they relate
The same relationships as text
- ad-service deploy prior revision revision 1847
- ad-service deploy current revision revision 1848. Deployment rolled forward to rev 1848 21m before the alert.
- revision 1848 owns pod ad-service pod
- ad-service pod runs ad-service runtime
- ad-service runtime loads config.yaml
- ad-service runtime binds :8000 kubelet probes
- CI/CD promotion promoted image revision 1848
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Node-level resource exhaustion (OOM) is killing the podRuled out
Read the container exit code and node-pressure events.
Exit code is 101 (application panic), not 137 (OOM). No node-pressure or OOMKilled events recorded. Fast 7-restart loop in 21m is a crash loop, not resource throttling.
Pod READY 0/1, CrashLoopBackOff, 7 restarts in 21m age.
$ kubectl get pod ad-service-767c6f45c-xgvkb -n prod
NAME READY STATUS RESTARTS AGE
ad-service-767c6f45c-xgvkb 0/1 CrashLoopBackOff 7 (4m10s) 21m
Tight crash loop with exponential backoff, not a slow resource squeeze.
Container process exits with code 101, not 137.
Process exited with code 101
101 = application panic. An OOM kill would be 137. Resource exhaustion ruled out.
Missing or broken config blocks startupRuled out
Read previous-container logs for a config load failure at /etc/ad-service/config.yaml.
Logs show the config loads cleanly (INFO Loading configuration, then INFO Initializing connection pool) before the FATAL. Config did not fail.
Config load and connection-pool init both log INFO before any error.
[INFO] Loading configuration from /etc/ad-service/config.yaml
[INFO] Initializing connection pool...
[FATAL] Assertion failed: Mutex::Fer ...
The panic is after a successful config load, so config is not the cause.
Liveness / readiness probe misconfiguration causes a false crash loopRuled out
Check probe config and events to see if probes are failing.
Probes fail with connection refused because port 8000 is never bound; the process exits 101 within ~1s of start before any probe fires. Probe config (delay=10s) is standard and worked for rev 1847. Probes confirm, not cause.
Probes fail dialing port 8000 with connection refused.
Liveness: http-get http://:8000/healthz delay=10s #failure=3
Readiness: http-get http://:8000/readyz delay=5s #failure=3
Warning Unhealthy kubelet Liveness probe failed:
dial tcp 10.0.47.12:8000: connect: connection refused
Port 8000 never binds, so probes can only fail. The crash precedes them.
A code defect in the deployed image crashes the runtime on startupConfirmed
Read previous-container logs to see if a code defect is crashing the application on startup.
Container logs show a fatal Mutex::Fer assertion at src/core/mutex_fer.rs:147, thread main panicked, exit 101, before binding port 8000. Every restart reproduces the same trace. Confirmed; correlate with the rollout next.
Runtime panics at a fixed source location during init on every start.
$ kubectl logs ad-service-767c6f45c-xgvkb -n prod --previous
[FATAL] Assertion failed: Mutex::Fer - cannot acquire lock on shared resource
at src/core/mutex_fer.rs:147
at src/main.rs:42
thread 'main' panicked at 'Mutex::Fer assertion failed', src/core/mutex_fer.rs:147
Process exited with code 101
Deterministic code-level panic at init, before any network or probe. A code defect in the image.
Deployment revision 1848 introduced the defective imageConfirmed
Diff rollout history against crash onset to identify any recent changes.
Crash appears exactly at rev 1848. The current crashing pod is owned by ReplicaSet ad-service-767c6f45c (rev 1848); rev 1847 ran 1d with 0 restarts and is now scaled to 0. The image jumped to a release candidate, v2.15.0-rc1.
rev 1848 introduced image v2.15.0-rc1, a release candidate.
REVISION CHANGE-CAUSE
1846 image update: ad-service:v2.14.0
1847 image update: ad-service:v2.14.1
1848 image update: ad-service:v2.15.0-rc1
The -rc1 tag is a release candidate promoted to prod; prior prod tags carried no rc suffix.
The crashing pod belongs to the rev-1848 ReplicaSet; rev 1847 is scaled to 0.
NAME DESIRED CURRENT READY AGE
ad-service-5c7e8b3a1 0 0 0 1d # rev 1847 (was stable)
ad-service-767c6f45c 1 1 0 21m # rev 1848 (crashing)
Crash onset tracks the rev 1848 rollout exactly; no config or infra change between 1847 and 1848.
Mutex::Fer assertion in v2.15.0-rc1 is the fatal defectRoot cause
Confirm the failing module by checking the crash log and verifying if it is new to the recent version.
The fatal Mutex::Fer assertion in ad-service:v2.15.0-rc1 (rev 1848) is the root cause. It aborts init at src/core/mutex_fer.rs:147 with exit 101 before port 8000 binds; probe failures and the restart loop are downstream effects. Rollback to rev 1847 (v2.14.1) restores service.
Panic is a Mutex::Fer lock-acquisition assertion at a fixed file and line.
[FATAL] Assertion failed: Mutex::Fer - cannot acquire lock on shared resource
at src/core/mutex_fer.rs:147
thread 'main' panicked, Process exited with code 101
A specific, reproducible assertion at mutex_fer.rs:147; absent in v2.14.x which ran with 0 restarts.
All 7 restarts fall inside the 21m since rev 1848; rev 1847 had 0.
rev 1847 (v2.14.1) restarts 0 stable 1d
rev 1848 (v2.15.0-rc1) restarts 7 age 21m
The defect arrived with v2.15.0-rc1; rolling back to 1847 is the fix.
Whether the rc tag reached prod via a pipeline gap or a manual changeNeeds access
Attempt to check the CI/CD audit log and Kubernetes audit log to see who applied rev 1848.
Outside this telemetry. The rc1 image reaching prod is consistent with an inadvertent promotion (no gate rejected the -rc tag), but confirming pipeline-gap vs manual change needs the audit trail.
No gate blocked the v2.15.0-rc1 tag, but the triggering actor is not in our telemetry.
Stops at the boundary and labels the gap rather than guessing the actor.
The panic occurs only on specific node architecturesRuled out
Check if the pod was scheduled on a node with a different architecture or OS.
The node pool is homogeneous and the panic reproduces locally on all environments.
Revision 1848 introduced an incompatible configuration changeRuled out
Diff the configuration associated with rev 1848 against rev 1847.
Configuration was unchanged between revisions.
External service dependency is down, causing a panicRuled out
Check startup logs for connection attempts to external services.
Logs show the panic occurs before any external network connections are attempted.
Final RCA
Mutex::Fer assertion in v2.15.0-rc1 is the fatal defect
The fatal Mutex::Fer assertion in ad-service:v2.15.0-rc1 (rev 1848) is the root cause. It aborts init at src/core/mutex_fer.rs:147 with exit 101 before port 8000 binds; probe failures and the restart loop are downstream effects. Rollback to rev 1847 (v2.14.1) restores service.
Remediation
# Rollback to the last stable revision
kubectl rollout undo deploy/ad-service -n prod --to-revision=1847
# Verify rollback
kubectl rollout status deploy/ad-service -n prod
kubectl get pods -n prod -l app=ad-service- Rollback immediately to revision 1847 (v2.14.1) to restore service.
- Investigate the Mutex::Fer assertion in v2.15.0-rc1: this is a code bug that needs to be fixed before re-deploying.
- Add deployment guardrails to prevent release candidate (rc) images from being deployed to production.
- Implement canary deployments or progressive rollouts to catch crash loops before full rollout.
- Consider adding a pre-deployment smoke test that verifies the container can start and pass health checks.
The defect shipped because a release candidate tag reached production with no gate to stop it, and the crash was only visible after rollout rather than before. A tag-pattern check that rejects anything matching *-rc* at the deployment step closes the gap that let v2.15.0-rc1 through, and a pre-deployment smoke test that confirms the container starts and passes health checks would have caught the Mutex::Fer panic before it reached seven restarts in prod. Canary or progressive rollout would have limited the blast radius to a fraction of traffic while the same evidence, exit code 101 and the fatal assertion in previous-container logs, surfaced.
Frequently asked questions
How was OOM ruled out as the cause of this CrashLoopBackOff?
The pod exited with code 101, an application panic, not code 137, which is what the kernel OOM killer produces. Zero node-pressure or OOMKilled events were recorded, and the 7 restarts happened in a tight 21-minute loop rather than a slow squeeze. Container logs showed a fatal Mutex::Fer assertion at src/core/mutex_fer.rs:147 on every restart, well before the process could bind its port.
What deployment change triggered the CrashLoopBackOff?
Deployment revision 1848 introduced image ad-service:v2.15.0-rc1, a release candidate, 21 minutes before the alert. The prior revision, 1847 (v2.14.1), had run stable for a day with zero restarts. Rollout history and ReplicaSet mapping both confirm the crashing pod belongs to revision 1848, and no configuration changed between the two revisions.
Did a liveness or readiness probe misconfiguration cause the crash loop?
No. Logs show the probes fail with connection refused because the process panics and exits before it ever binds port 8000, so the probes never have a chance to succeed. Probe configuration (delay=10s) was unchanged from revision 1847, which ran without issue, and the fatal assertion occurs within about a second of startup, before any probe fires.
How was ad-service recovered after the CrashLoopBackOff?
Roll back to revision 1847 (v2.14.1), which ran stable for a day with zero restarts, using kubectl rollout undo. Investigating and fixing the Mutex::Fer assertion in v2.15.0-rc1 is required before it can be redeployed. Longer term, add a guardrail that blocks release candidate image tags from reaching production and a pre-deployment smoke test that confirms the container starts and passes health checks.