Alert info
- Severity
- Critical
- Detected by
- Grafana v10.1.2
- Alert
- Image Pull / CrashLoopBackOff on trino-coordinator and trino-worker pods
- Time
- 2026-02-25 23:30 IST
- Service
- trino-coordinator / trino-worker (analytics)
Grafana fired an Image Pull / CrashLoopBackOff alert on the analytics Trino pods. Investigate the cause.
Trino coordinator and worker pods entered CrashLoopBackOff during cluster-wide resource contention (insufficient memory/cpu). The Image Pull portion of the Grafana alert was a false positive: pulls of trinodb/trino:475 succeeded. A manual rollout restart replaced the failing ReplicaSets and recovered service.
- Status
- Resolved
- Recovery
- Rollout restart
- Image Pull
- False positive
Entities identified
Infrastructure
analytics nodes
cluster capacity
Analytics node pool.
Service
trino-coordinator
analytics
Coordinator Deployment for Trino.
trino-worker
analytics
Worker Deployment for Trino.
Data
memory
node resource
Node memory capacity.
cpu
node resource
Node CPU capacity.
External
trinodb/trino:475
container image
Pinned container image
trinodb/trino:475.
Identity
Vault Secrets Operator
restart trigger
Vault Secrets Operator.
Gateway
Grafana
alerting
Grafana alerting system.
How they relate
The same relationships as text
- Grafana alerts on trino-coordinator
- Grafana alerts on trino-worker
- trino-coordinator pulls trinodb/trino:475. Coordinator pulls the container image.
- trino-worker pulls trinodb/trino:475. Worker pulls the container image.
- trino-coordinator scheduled on analytics nodes
- trino-worker scheduled on analytics nodes
- analytics nodes exhausted memory. Node pool depends on memory availability.
- analytics nodes exhausted cpu. Node pool depends on CPU availability.
- analytics nodes evicts / fails to schedule trino-worker. Node pool schedules worker pods.
- Vault Secrets Operator restarts trino-worker. VSO can trigger pod restarts.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Image pull failed for trinodb/trino:475Ruled out
Query Kubernetes events for ErrImagePull or ImagePullBackOff.
Events show a Pulled event confirming a successful pull of trinodb/trino:475. No ErrImagePull or ImagePullBackOff. The Image Pull leg of the alert is a false positive, likely a transient pull during scheduling churn.
Pull events for trinodb/trino:475 during the incident window.
REASON OBJECT MESSAGE
Pulled pod/trino-worker-58f7b5d9dd-qnv9t Successfully pulled trinodb/trino:475
(no ErrImagePull / ImagePullBackOff events present)
Image was available and pulled. The Image Pull alert leg is not corroborated.
Missing JVM / connector config crashed Trino on startup (sibling case)Ruled out
Compare behaviors with known config issues to see if they reproduce on the same image and configuration.
The crash resolved via a plain rollout restart at 21:01 UTC with no config change. A config-driven startup failure would reproduce on the same image. Events also show cluster-wide FailedScheduling across workloads, not a single-service startup fault.
Rollout restart used the identical image and config; pods came up healthy.
kubectl.kubernetes.io/restartedAt: 2026-02-25T21:01:53Z
new pods: trino-coordinator-749b687f9d 1/1 Running
trino-worker-dc88c89cb 4/4 Running
Same image/config now stable. A config-level crash would persist; this did not.
Vault Secrets Operator restart alone caused the crash loopRuled out
Evaluate if the Vault Secrets Operator restart timing accounts for the CrashLoopBackOff.
VSO restarted both Deployments at 18:32:15Z, which adds churn, but a clean secret-rotation restart does not by itself produce a multi-hour crash loop with FailedScheduling. It is a contributing factor in timing, not the cause. Ruled out as sole cause.
VSO triggered a restart on both worker and coordinator at the same instant.
vso.secrets.hashicorp.com/restartedAt: 2026-02-25T18:32:15Z (worker)
vso.secrets.hashicorp.com/restartedAt: 2026-02-25T18:32:15Z (coordinator)
Overlaps the crash window, so it adds churn, but cannot explain FailedScheduling. Contributing, not causal.
Cluster-wide resource pressure destabilized the Trino podsConfirmed
Analyze cluster events for scheduling failures, evictions, and probe status.
Events show FailedScheduling citing Insufficient memory and Insufficient cpu, plus pod evictions, concurrent with Unhealthy probe events on the trino pods. The crash loop sits inside a contended cluster, not an image or config fault. Confirmed; narrow to the exact resource.
FailedScheduling and Evicted events across the incident window.
TYPE REASON MESSAGE
Warning FailedScheduling 0/N nodes: Insufficient memory, Insufficient cpu, node(s) had taints / not ready
Warning Evicted pod evicted (reason: Underutilized), rescheduling
The cluster is under memory/cpu pressure during the window; pods cannot stabilize.
Insufficient memory/cpu drove probe failures and BackOffConfirmed
Correlate FailedScheduling events with probe failures.
FailedScheduling explicitly cites Insufficient memory and Insufficient cpu. Liveness saw connection refused and readiness saw CONTAINER_EXITED / Server is starting, and BackOff fired on the old RS pods. The pods could not come up under starvation.
Probe failures and BackOff on the alert-hash pods during pressure.
Unhealthy Liveness probe failed: connection refused
Unhealthy Readiness probe failed: CONTAINER_EXITED / Server is starting
BackOff trino-worker-58f7b5d9dd-qnv9t
BackOff trino-coordinator-68bf449659-dvr9m
Process never reaches a listening state under starvation, so probes fail and BackOff loops.
Alert-hash RSes are the ones that crash-looped; they are now scaled to 0.
trino-coordinator-749b687f9d 1 1 1 7m4s (active)
trino-worker-dc88c89cb 4 4 4 7m8s (active)
trino-coordinator-68bf449659 0 0 0 60m (matches alert hash)
trino-worker-58f7b5d9dd 0 0 0 60m (matches alert hash)
Confirms the crashing pods were the alert-named RSes, replaced by the rollout restart.
No resource requests/limits left Trino BestEffort and first to be evictedRoot cause
Inspect the Trino Deployments for resource requests and limits.
With no resource requests/limits, the Trino pods are BestEffort and have no scheduling-priority guarantee, so under insufficient memory/cpu they are evicted first and cannot be rescheduled. That is what turned cluster pressure into a sustained CrashLoopBackOff. Root cause. Recovery was a manual kubectl rollout restart at 21:01 UTC onto freed capacity.
No requests/limits set, so QoS is BestEffort.
Containers:
trino:
Limits: <none>
Requests: <none>
QoS Class: BestEffort
BestEffort pods are evicted first under node pressure and have no scheduling guarantee. This is why Trino crash-looped.
Manual restart at 21:01 UTC replaced the failing RSes onto available capacity.
kubectl.kubernetes.io/restartedAt: 2026-02-25T21:01:49Z (worker)
kubectl.kubernetes.io/restartedAt: 2026-02-25T21:01:53Z (coordinator)
-> new RSes dc88c89cb / 749b687f9d, old RSes scaled to 0
Restarting onto freed capacity recovered service without any config change, consistent with a resource-driven cause.
Exact crash stacktraces / exit codesNeeds access
Would require the original pod logs or persisted ELK/Loki history.
The alert pods are gone (NotFound) and the ELK fetch failed, so exact exit codes are unrecoverable. The investigation stops at the resource evidence rather than guessing an app-level co-failure.
Crash logs for the alert pods cannot be retrieved post-replacement.
Error from server (NotFound): pods "trino-coordinator-68bf449659-thq8v" not found
Error from server (NotFound): pods "trino-worker-58f7b5d9dd-92mnn" not found
Stacktraces are gone; an app-level co-failure cannot be fully excluded. Needs persisted logs.
Node disk pressure caused pod evictionsRuled out
Check the node conditions for DiskPressure.
No DiskPressure was observed on the nodes; the resource pressure was limited to memory and CPU.
Final RCA
No resource requests/limits left Trino BestEffort and first to be evicted
With no resource requests/limits, the Trino pods are BestEffort and have no scheduling-priority guarantee, so under insufficient memory/cpu they are evicted first and cannot be rescheduled. That is what turned cluster pressure into a sustained CrashLoopBackOff. Root cause. Recovery was a manual kubectl rollout restart at 21:01 UTC onto freed capacity.
Remediation
# Set resource requests/limits to ensure scheduling priority
kubectl set resources deploy/trino-coordinator -n analytics \
--requests=cpu=2,memory=8Gi --limits=cpu=4,memory=16Gi
# Add PodDisruptionBudget to prevent mass evictions
kubectl apply -f - <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: trino-worker-pdb
namespace: analytics
spec:
minAvailable: 2
selector:
matchLabels:
app: trino-worker
EOF- Set appropriate resource requests and limits for Trino coordinator and workers to ensure scheduling priority during contention.
- Add PodDisruptionBudgets (PDB) for Trino workloads to prevent mass evictions during node pressure.
- Consider using PriorityClass to give analytics workloads higher scheduling priority than batch jobs.
- Increase liveness probe initialDelaySeconds and failureThreshold for Trino, as it has a slow startup.
- Set up persistent log aggregation (ELK/Loki) so crash logs are preserved even after pod replacement.
- Investigate the cluster capacity planning: FailedScheduling events suggest the cluster is running at or near capacity.
- Review the Grafana alert rule: the Image Pull component fired as a false positive and should be tuned.
Trino was scheduled without resource requests or limits, so it had no priority over anything else contending for the same nodes and was among the first evicted once memory and CPU ran short. Setting requests and limits on the coordinator and worker Deployments, backed by a PodDisruptionBudget and a PriorityClass appropriate to an analytics workload, gives Trino a scheduling guarantee that batch jobs do not have. Persistent log aggregation closes the other gap this incident exposed: once the crashing pods were replaced, their logs were gone, and the investigation had to conclude on cluster events alone rather than exact stack traces.
Frequently asked questions
Why did Trino pods enter CrashLoopBackOff if the image pulled successfully?
Kubernetes events confirmed trinodb/trino:475 pulled successfully with no ErrImagePull or ImagePullBackOff, so the Image Pull portion of the alert was a false positive. The real cause was cluster-wide FailedScheduling citing insufficient memory and CPU, which triggered liveness and readiness probe failures and put the pods into BackOff restart loops.
What actually made Trino unstable during the resource pressure window?
The Trino coordinator and worker Deployments had no resource requests or limits, so Kubernetes scheduled them as BestEffort pods with no scheduling-priority guarantee. Under cluster-wide FailedScheduling for insufficient memory and CPU, BestEffort pods are evicted and destabilized first, which turned normal contention into a sustained CrashLoopBackOff.
How was the Trino CrashLoopBackOff resolved?
A manual kubectl rollout restart executed at 21:01 UTC replaced the crashing ReplicaSets with new ones scheduled onto freed cluster capacity. No configuration or image change was involved. The old ReplicaSets scaled to zero and all new Trino pods came up Running, confirming the fault was resource pressure rather than a code or config defect.
Is this the same root cause as the Trino BigQuery JVM argument crash loop?
No. This incident is cluster-wide resource pressure with no resource limits set on the Trino Deployments, confirmed by FailedScheduling and eviction events. The sibling investigation, Trino CrashLoopBackOff from a missing BigQuery JVM argument, is a deterministic Exit Code 100 config validation failure that reproduces regardless of cluster load. Same alert signature, different root cause.