Trino CrashLoopBackOff from a missing BigQuery connector JVM argument

Alert info

Severity
Critical
Detected by
Grafana v10.1.2
Alert
Image Pull / CrashLoopBackOff on trino-coordinator and trino-worker pods
Time
2026-02-25 20:43 UTC
Service
trino-coordinator / trino-worker (analytics)

Coordinator restarting and 3 of 4 workers in CrashLoopBackOff. Investigate the cause.

Trino pods crash on startup (Exit Code 100) because the BigQuery connector requires --add-opens=java.base/java.nio=ALL-UNNAMED, which is missing from JVM config. Image Pull alert was a false positive.

Exit Code
100
Affected Pods
4/5
Root Cause
Missing JVM arg

Entities identified

Infrastructure

  • Grafana

    alerting

    Grafana alerting system.

  • rollout 20:08:47

    helm / argocd

    Helm rollout deployment process.

  • trinodb/trino:475

    container image

    Container image trinodb/trino:475.

Service

  • trino-coordinator

    analytics

    Trino coordinator pod.

  • trino-worker

    analytics

    Trino worker pods.

Data

  • jvm.config

    ConfigMap

    JVM configuration maps.

  • news_bq.properties

    catalog ConfigMap

    Trino catalog configuration.

External

  • BigQuery connector

    trino plugin

    BigQuery connector for Trino.

How they relate

alerts onalerts oncreated RScreated RSpulledloadsenablesrequires argGrafanaalertingrollout 20:08:47helm / argocdtrinodb/trino:475container imagetrino-coordinatoranalyticstrino-workeranalyticsnews_bq.propertiescatalog ConfigMapBigQuery connectortrino pluginjvm.configConfigMap
The same relationships as text
  • Grafana alerts on trino-coordinator
  • Grafana alerts on trino-worker
  • rollout 20:08:47 created RS trino-coordinator. Rollout creates the coordinator ReplicaSet.
  • rollout 20:08:47 created RS trino-worker. Rollout creates the worker ReplicaSet.
  • trinodb/trino:475 pulled trino-worker. Worker pulls the container image.
  • trino-worker mounts jvm.config
  • trino-coordinator mounts jvm.config
  • trino-coordinator loads news_bq.properties
  • news_bq.properties enables BigQuery connector
  • BigQuery connector requires arg jvm.config. Connector defines JVM requirements.

Hypotheses tree

Ruled-out branches are collapsed. Open any one to read the check that eliminated it.

ImagePull failure prevented the container from startingRuled out

Review pod events and exit states to verify if an image pull failure occurred.

Image pulled successfully in 127ms. The container starts and then exits with code 100, which is Trino's config validation exit code, not an image-layer error. The Image Pull alert label is a false positive.

Pod describe, from kubectl describe pod

Image pulled cleanly; last state Terminated (Error) with Exit Code 100.

Image: trinodb/trino:475 (pulled successfully)
Last State: Terminated (Error)
Exit Code: 100
Events:
  Normal  Pulled  Successfully pulled image "trinodb/trino:475" in 127ms

The container starts then exits. Exit 100 is config validation, not ImagePull. Ruled out.

Resource pressure (CPU/memory) caused OOM kill or evictionRuled out

Inspect pod exit codes and node events for resource exhaustion signatures.

No OOMKilled exit code and no resource-quota or node-pressure events anywhere in the data. Exit Code 100 is explicitly Trino config validation, a deterministic startup failure.

Exit code + events, from kubectl describe / get events

Every crashing pod exits 100, not 137 (OOM); no eviction or node-pressure events.

exit code      100   (config validation)
OOMKilled      no
node pressure  none
quota events   none

Failure is deterministic config validation, not resource pressure. Ruled out.

Cluster-wide vs intermittent: is only one worker affected?Ruled out

Examine the state of all Trino pods across the cluster to see if the issue is isolated.

The surviving worker bkgz6 is 33m old, predating the 20:08:47Z rollout, so it runs the previous ReplicaSet. Every pod from the new ReplicaSet crashes identically. Not intermittent: failure is deterministic and cluster-wide.

Cluster pod state, from kubectl get pods -n analytics

Coordinator + 3 new workers crashing; only the pre-rollout worker is healthy.

trino-coordinator-68bf449659-thq8v   0/1  Running           2  2m7s
trino-worker-58f7b5d9dd-92mnn        0/1  CrashLoopBackOff  7  19m
trino-worker-58f7b5d9dd-bkgz6        1/1  Running           0  33m
trino-worker-58f7b5d9dd-qnv9t        0/1  CrashLoopBackOff  6  11m
trino-worker-58f7b5d9dd-r6tx6        0/1  CrashLoopBackOff  6  11m

bkgz6 predates the rollout; all new-RS pods crash the same way. Cluster-wide, deterministic.

Trino exits on startup config validation (Exit Code 100)Confirmed

Extract and analyze the previous-container logs from a crashed worker.

Worker log at 20:42:15Z shows io.trino.server.Server reporting configuration errors: the BigQuery connector requires an additional JVM argument. This is a config validation failure, not a runtime crash. Confirmed; trace which arg and where it is missing.

Worker startup log, from kubectl logs --previous

Trino's config validator prints a fatal error naming the BigQuery connector's JVM-arg requirement.

2026-02-25T20:42:15.224Z ERROR main io.trino.server.Server Configuration errors:
1) Error: Connector 'bigquery' requires additional JVM argument(s).
   Please add the following to the JVM configuration:
   '--add-opens=java.base/java.nio=ALL-UNNAMED'

Exit 100 is config validation failing on a missing JVM arg the BigQuery connector demands. Confirmed.

BigQuery connector is enabled and demands a JVM arg that is absentConfirmed

Audit the catalog and JVM ConfigMaps for connector configurations.

news_bq.properties enables connector.name=bigquery, so Trino runs the connector's startup JVM-arg check. The worker and coordinator jvm.config ConfigMaps list only -Xmx8G, -XX:+UseG1GC, -XX:+ExitOnOutOfMemoryError. The required --add-opens arg is absent.

Catalog enablement, from kubectl get configmap (catalog)

BigQuery connector is enabled via news_bq.properties, activating the JVM-arg requirement.

news_bq.properties:
  connector.name=bigquery
  project-id=********
  credentials-key=${ENV:bigquery_credentials_key}

Enabling bigquery is what triggers Trino's --add-opens check at startup.

JVM config gap, from kubectl get configmap trino-worker

jvm.config carries heap/GC flags only; the required --add-opens is not present.

jvm.config: |
  -Xmx8G
  -XX:+UseG1GC
  -XX:+ExitOnOutOfMemoryError
  # --add-opens=java.base/java.nio=ALL-UNNAMED  NOT present

Connector enabled but its mandatory JVM arg is missing from the ConfigMap. This is the gap.

Rollout at 20:08:47Z applied JVM ConfigMaps that omit --add-opensRoot cause

Correlate deployment history with the onset of the crash loop.

Both ReplicaSets were created at 20:08:47Z with jvm.config ConfigMaps that omit --add-opens=java.base/java.nio=ALL-UNNAMED while the BigQuery catalog stays enabled. Pods fail validation, exit 100, and CrashLoop. The coordinator log at 20:48:55Z shows the same error. The single missing JVM arg is the root cause; adding it to jvm.config (Helm values preferred) clears the crash. Image Pull was a false positive.

Change correlation, from kubectl get rs

Both active ReplicaSets created at the rollout timestamp; pods began crashing immediately after.

trino-coordinator-68bf449659  CREATED 2026-02-25T20:08:47Z  REPLICAS 1
trino-worker-58f7b5d9dd       CREATED 2026-02-25T20:08:47Z  REPLICAS 4
worker lost minimum availability  20:35:54Z
alert fired                       20:43:31Z

Rollout at 20:08:47Z is the change point that applied the JVM ConfigMaps missing the arg.

Coordinator confirms same error, from kubectl logs --previous (coordinator)

Coordinator prints the identical BigQuery JVM-arg config error, proving a cluster-wide ConfigMap gap.

2026-02-25T20:48:55.349Z ERROR main io.trino.server.Server Configuration errors:
1) Error: Connector 'bigquery' requires additional JVM argument(s).
   '--add-opens=java.base/java.nio=ALL-UNNAMED'

Same missing arg on coordinator and workers. Single fix: add the arg to jvm.config. Root cause.

The BigQuery connector credentials are invalidRuled out

Check the Trino logs for authentication errors.

Trino does not reach the authentication phase; it fails earlier during JVM argument validation.

Final RCA

Rollout at 20:08:47Z applied JVM ConfigMaps that omit --add-opens

Both ReplicaSets were created at 20:08:47Z with jvm.config ConfigMaps that omit --add-opens=java.base/java.nio=ALL-UNNAMED while the BigQuery catalog stays enabled. Pods fail validation, exit 100, and CrashLoop. The coordinator log at 20:48:55Z shows the same error. The single missing JVM arg is the root cause; adding it to jvm.config (Helm values preferred) clears the crash. Image Pull was a false positive.

Remediation

# Option 1: Patch ConfigMaps directly
kubectl get configmap trino-worker -n analytics -o yaml | \
  sed 's/jvm.config: |/jvm.config: |\n  --add-opens=java.base\/java.nio=ALL-UNNAMED/' | \
  kubectl apply -f -

# Repeat for coordinator ConfigMap
kubectl get configmap trino-coordinator -n analytics -o yaml | \
  sed 's/jvm.config: |/jvm.config: |\n  --add-opens=java.base\/java.nio=ALL-UNNAMED/' | \
  kubectl apply -f -

# Option 2 (preferred): Update Helm values and redeploy
# In values.yaml under jvm.config, add:
#   --add-opens=java.base/java.nio=ALL-UNNAMED

# Restart pods to pick up config change
kubectl rollout restart deployment/trino-coordinator -n analytics
kubectl rollout restart deployment/trino-worker -n analytics
  • Add --add-opens=java.base/java.nio=ALL-UNNAMED to Trino JVM configuration (Helm values preferred over direct ConfigMap edit).
  • Add a CI/CD validation step that checks connector requirements against JVM configuration before deploying.
  • Review Trino upgrade notes when bumping versions: connector JVM requirements change across versions.
  • Consider adding a Helm chart test or init container that validates Trino configuration before the main container starts.
  • Tune the Grafana alert to distinguish between ImagePull failures and application startup failures (Exit Code 100 != ImagePull).

The BigQuery connector's JVM-argument requirement was added to the catalog configuration without a matching update to jvm.config, so the gap sat invisible until the next rollout applied it cluster-wide. Moving jvm.config into the same Helm values file that enables a connector, so the two change together, removes the chance of shipping one without the other. A CI validation step that checks each enabled connector's documented JVM requirements against the rendered jvm.config, and a Helm chart test or init container that runs Trino's own config validation before the main container starts, would have caught this before it ever reached the cluster.

Frequently asked questions

Why do Trino pods exit with Exit Code 100 on startup?

Exit Code 100 is Trino Configuration validation failing before the server starts, not a runtime crash or an image pull problem. Worker and coordinator logs both printed the identical fatal error: Connector bigquery requires additional JVM argument(s), naming add-opens=java.base/java.nio=ALL-UNNAMED as the missing flag. The container starts, fails validation, and exits deterministically every time.

Is this the same root cause as the Trino resource pressure crash loop?

This is a different incident. Trino CrashLoopBackOff from cluster resource pressure shows FailedScheduling and eviction events with no config error, and it resolved via a plain rollout restart with no config change. This incident exits with a deterministic Exit Code 100 config validation error tied to the BigQuery connector, and it reproduces on every pod regardless of cluster load.

How do you fix the missing add-opens JVM argument for Trino's BigQuery connector?

Add add-opens=java.base/java.nio=ALL-UNNAMED to the jvm.config ConfigMap for both trino-worker and trino-coordinator, preferably by updating the Helm values that generate them so the fix survives the next rollout. Restarting the Deployments after the ConfigMap change lets Trino pass startup validation and stay running.

Was the Image Pull alert on the Trino pods a real failure?

No. kubectl describe pod showed the image trinodb/trino:475 pulled successfully in 127ms with no ErrImagePull or ImagePullBackOff events. The container starts normally and then exits with Exit Code 100, which is a configuration validation failure inside Trino, not an image-layer problem.

Related investigations