Alert info
- Severity
- Critical
- Detected by
- Grafana v10.1.2
- Alert
- Image Pull / CrashLoopBackOff on trino-coordinator and trino-worker pods
- Time
- 2026-02-25 20:43 UTC
- Service
- trino-coordinator / trino-worker (analytics)
Coordinator restarting and 3 of 4 workers in CrashLoopBackOff. Investigate the cause.
Trino pods crash on startup (Exit Code 100) because the BigQuery connector requires --add-opens=java.base/java.nio=ALL-UNNAMED, which is missing from JVM config. Image Pull alert was a false positive.
- Exit Code
- 100
- Affected Pods
- 4/5
- Root Cause
- Missing JVM arg
Entities identified
Infrastructure
Grafana
alerting
Grafana alerting system.
rollout 20:08:47
helm / argocd
Helm rollout deployment process.
trinodb/trino:475
container image
Container image
trinodb/trino:475.
Service
trino-coordinator
analytics
Trino coordinator pod.
trino-worker
analytics
Trino worker pods.
Data
jvm.config
ConfigMap
JVM configuration maps.
news_bq.properties
catalog ConfigMap
Trino catalog configuration.
External
BigQuery connector
trino plugin
BigQuery connector for Trino.
How they relate
The same relationships as text
- Grafana alerts on trino-coordinator
- Grafana alerts on trino-worker
- rollout 20:08:47 created RS trino-coordinator. Rollout creates the coordinator ReplicaSet.
- rollout 20:08:47 created RS trino-worker. Rollout creates the worker ReplicaSet.
- trinodb/trino:475 pulled trino-worker. Worker pulls the container image.
- trino-worker mounts jvm.config
- trino-coordinator mounts jvm.config
- trino-coordinator loads news_bq.properties
- news_bq.properties enables BigQuery connector
- BigQuery connector requires arg jvm.config. Connector defines JVM requirements.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
ImagePull failure prevented the container from startingRuled out
Review pod events and exit states to verify if an image pull failure occurred.
Image pulled successfully in 127ms. The container starts and then exits with code 100, which is Trino's config validation exit code, not an image-layer error. The Image Pull alert label is a false positive.
Image pulled cleanly; last state Terminated (Error) with Exit Code 100.
Image: trinodb/trino:475 (pulled successfully)
Last State: Terminated (Error)
Exit Code: 100
Events:
Normal Pulled Successfully pulled image "trinodb/trino:475" in 127ms
The container starts then exits. Exit 100 is config validation, not ImagePull. Ruled out.
Resource pressure (CPU/memory) caused OOM kill or evictionRuled out
Inspect pod exit codes and node events for resource exhaustion signatures.
No OOMKilled exit code and no resource-quota or node-pressure events anywhere in the data. Exit Code 100 is explicitly Trino config validation, a deterministic startup failure.
Every crashing pod exits 100, not 137 (OOM); no eviction or node-pressure events.
exit code 100 (config validation)
OOMKilled no
node pressure none
quota events none
Failure is deterministic config validation, not resource pressure. Ruled out.
Cluster-wide vs intermittent: is only one worker affected?Ruled out
Examine the state of all Trino pods across the cluster to see if the issue is isolated.
The surviving worker bkgz6 is 33m old, predating the 20:08:47Z rollout, so it runs the previous ReplicaSet. Every pod from the new ReplicaSet crashes identically. Not intermittent: failure is deterministic and cluster-wide.
Coordinator + 3 new workers crashing; only the pre-rollout worker is healthy.
trino-coordinator-68bf449659-thq8v 0/1 Running 2 2m7s
trino-worker-58f7b5d9dd-92mnn 0/1 CrashLoopBackOff 7 19m
trino-worker-58f7b5d9dd-bkgz6 1/1 Running 0 33m
trino-worker-58f7b5d9dd-qnv9t 0/1 CrashLoopBackOff 6 11m
trino-worker-58f7b5d9dd-r6tx6 0/1 CrashLoopBackOff 6 11m
bkgz6 predates the rollout; all new-RS pods crash the same way. Cluster-wide, deterministic.
Trino exits on startup config validation (Exit Code 100)Confirmed
Extract and analyze the previous-container logs from a crashed worker.
Worker log at 20:42:15Z shows io.trino.server.Server reporting configuration errors: the BigQuery connector requires an additional JVM argument. This is a config validation failure, not a runtime crash. Confirmed; trace which arg and where it is missing.
Trino's config validator prints a fatal error naming the BigQuery connector's JVM-arg requirement.
2026-02-25T20:42:15.224Z ERROR main io.trino.server.Server Configuration errors:
1) Error: Connector 'bigquery' requires additional JVM argument(s).
Please add the following to the JVM configuration:
'--add-opens=java.base/java.nio=ALL-UNNAMED'
Exit 100 is config validation failing on a missing JVM arg the BigQuery connector demands. Confirmed.
BigQuery connector is enabled and demands a JVM arg that is absentConfirmed
Audit the catalog and JVM ConfigMaps for connector configurations.
news_bq.properties enables connector.name=bigquery, so Trino runs the connector's startup JVM-arg check. The worker and coordinator jvm.config ConfigMaps list only -Xmx8G, -XX:+UseG1GC, -XX:+ExitOnOutOfMemoryError. The required --add-opens arg is absent.
BigQuery connector is enabled via news_bq.properties, activating the JVM-arg requirement.
news_bq.properties:
connector.name=bigquery
project-id=********
credentials-key=${ENV:bigquery_credentials_key}
Enabling bigquery is what triggers Trino's --add-opens check at startup.
jvm.config carries heap/GC flags only; the required --add-opens is not present.
jvm.config: |
-Xmx8G
-XX:+UseG1GC
-XX:+ExitOnOutOfMemoryError
# --add-opens=java.base/java.nio=ALL-UNNAMED NOT present
Connector enabled but its mandatory JVM arg is missing from the ConfigMap. This is the gap.
Rollout at 20:08:47Z applied JVM ConfigMaps that omit --add-opensRoot cause
Correlate deployment history with the onset of the crash loop.
Both ReplicaSets were created at 20:08:47Z with jvm.config ConfigMaps that omit --add-opens=java.base/java.nio=ALL-UNNAMED while the BigQuery catalog stays enabled. Pods fail validation, exit 100, and CrashLoop. The coordinator log at 20:48:55Z shows the same error. The single missing JVM arg is the root cause; adding it to jvm.config (Helm values preferred) clears the crash. Image Pull was a false positive.
Both active ReplicaSets created at the rollout timestamp; pods began crashing immediately after.
trino-coordinator-68bf449659 CREATED 2026-02-25T20:08:47Z REPLICAS 1
trino-worker-58f7b5d9dd CREATED 2026-02-25T20:08:47Z REPLICAS 4
worker lost minimum availability 20:35:54Z
alert fired 20:43:31Z
Rollout at 20:08:47Z is the change point that applied the JVM ConfigMaps missing the arg.
Coordinator prints the identical BigQuery JVM-arg config error, proving a cluster-wide ConfigMap gap.
2026-02-25T20:48:55.349Z ERROR main io.trino.server.Server Configuration errors:
1) Error: Connector 'bigquery' requires additional JVM argument(s).
'--add-opens=java.base/java.nio=ALL-UNNAMED'
Same missing arg on coordinator and workers. Single fix: add the arg to jvm.config. Root cause.
The BigQuery connector credentials are invalidRuled out
Check the Trino logs for authentication errors.
Trino does not reach the authentication phase; it fails earlier during JVM argument validation.
Final RCA
Rollout at 20:08:47Z applied JVM ConfigMaps that omit --add-opens
Both ReplicaSets were created at 20:08:47Z with jvm.config ConfigMaps that omit --add-opens=java.base/java.nio=ALL-UNNAMED while the BigQuery catalog stays enabled. Pods fail validation, exit 100, and CrashLoop. The coordinator log at 20:48:55Z shows the same error. The single missing JVM arg is the root cause; adding it to jvm.config (Helm values preferred) clears the crash. Image Pull was a false positive.
Remediation
# Option 1: Patch ConfigMaps directly
kubectl get configmap trino-worker -n analytics -o yaml | \
sed 's/jvm.config: |/jvm.config: |\n --add-opens=java.base\/java.nio=ALL-UNNAMED/' | \
kubectl apply -f -
# Repeat for coordinator ConfigMap
kubectl get configmap trino-coordinator -n analytics -o yaml | \
sed 's/jvm.config: |/jvm.config: |\n --add-opens=java.base\/java.nio=ALL-UNNAMED/' | \
kubectl apply -f -
# Option 2 (preferred): Update Helm values and redeploy
# In values.yaml under jvm.config, add:
# --add-opens=java.base/java.nio=ALL-UNNAMED
# Restart pods to pick up config change
kubectl rollout restart deployment/trino-coordinator -n analytics
kubectl rollout restart deployment/trino-worker -n analytics- Add
--add-opens=java.base/java.nio=ALL-UNNAMEDto Trino JVM configuration (Helm values preferred over direct ConfigMap edit). - Add a CI/CD validation step that checks connector requirements against JVM configuration before deploying.
- Review Trino upgrade notes when bumping versions: connector JVM requirements change across versions.
- Consider adding a Helm chart test or init container that validates Trino configuration before the main container starts.
- Tune the Grafana alert to distinguish between ImagePull failures and application startup failures (Exit Code 100 != ImagePull).
The BigQuery connector's JVM-argument requirement was added to the catalog configuration without a matching update to jvm.config, so the gap sat invisible until the next rollout applied it cluster-wide. Moving jvm.config into the same Helm values file that enables a connector, so the two change together, removes the chance of shipping one without the other. A CI validation step that checks each enabled connector's documented JVM requirements against the rendered jvm.config, and a Helm chart test or init container that runs Trino's own config validation before the main container starts, would have caught this before it ever reached the cluster.
Frequently asked questions
Why do Trino pods exit with Exit Code 100 on startup?
Exit Code 100 is Trino Configuration validation failing before the server starts, not a runtime crash or an image pull problem. Worker and coordinator logs both printed the identical fatal error: Connector bigquery requires additional JVM argument(s), naming add-opens=java.base/java.nio=ALL-UNNAMED as the missing flag. The container starts, fails validation, and exits deterministically every time.
Is this the same root cause as the Trino resource pressure crash loop?
This is a different incident. Trino CrashLoopBackOff from cluster resource pressure shows FailedScheduling and eviction events with no config error, and it resolved via a plain rollout restart with no config change. This incident exits with a deterministic Exit Code 100 config validation error tied to the BigQuery connector, and it reproduces on every pod regardless of cluster load.
How do you fix the missing add-opens JVM argument for Trino's BigQuery connector?
Add add-opens=java.base/java.nio=ALL-UNNAMED to the jvm.config ConfigMap for both trino-worker and trino-coordinator, preferably by updating the Helm values that generate them so the fix survives the next rollout. Restarting the Deployments after the ConfigMap change lets Trino pass startup validation and stay running.
Was the Image Pull alert on the Trino pods a real failure?
No. kubectl describe pod showed the image trinodb/trino:475 pulled successfully in 127ms with no ErrImagePull or ImagePullBackOff events. The container starts normally and then exits with Exit Code 100, which is a configuration validation failure inside Trino, not an image-layer problem.