Alert info
- Severity
- Critical
- Detected by
- Kubernetes
- Alert
- Container adhoc-worker stuck in CreateContainerConfigError
- Time
- 2026-03-17 05:13 IST
- Service
- analytics-worker (prod)
analytics-worker-0 cannot start its adhoc-worker container. It is stuck in CreateContainerConfigError. Investigate the cause.
Container stuck in CreateContainerConfigError for 22+ minutes because a required Kubernetes Secret was missing from the namespace. Image pulls fine: config assembly blocked.
- Failures
- 71
- Duration
- 22 min
- Restarts
- 0
Entities identified
Infrastructure
prod
namespace
Production namespace containing analytics-worker pods.
kubelet
node agent
Node agent responsible for assembling the container config and starting the pod.
Service
analytics-worker-0
pod
Pod owning the adhoc-worker container. State Waiting (CreateContainerConfigError).
adhoc-worker
container
The stuck container waiting since 05:13 IST.
Data
analytics-config
Secret (missing)
A Secret referenced by the container's environment configuration.
External
ECR registry
prod/analytics
Private ECR registry in
ap-south-1hosting the container image.
How they relate
The same relationships as text
- kubelet manages analytics-worker-0. kubelet runs the pod on its node and assembles each container's config before start.
- analytics-worker-0 runs adhoc-worker
- kubelet pulls image ECR registry. Kubelet pulls the image from the registry.
- adhoc-worker envFrom secretRef analytics-config. Container expects environment variables from this Secret.
- analytics-config expected in prod. The Secret is expected to reside in the prod namespace.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Image pull failure (ImagePullBackOff)Ruled out
Read the pod event log to verify if the image pull is succeeding or failing.
Events show the image pulls successfully in 127ms (893 MB), repeatedly, including a pull at 2m37s. No ErrImagePull.
Pull succeeds in 127ms and is retried cleanly; no backoff.
LAST SEEN TYPE REASON MESSAGE
17m Normal Pulled Successfully pulled image in 127ms (893 MB)
2m37s Normal Pulling Pulling image "***:2025-06-03-v10"
The image is fine; the failure is after the pull, during config assembly.
Container crashes on startup (CrashLoopBackOff)Ruled out
Check container State and Restart Count to see if the process is repeatedly crashing.
State is Waiting with Reason CreateContainerConfigError, not Terminated, and Restart Count is 0. The process never executed.
Waiting/CreateContainerConfigError with zero restarts.
State: Waiting
Reason: CreateContainerConfigError
Ready: False
Restart Count: 0
Restart Count 0 means the container never ran; this is not a crash loop.
ECR authentication or registry failureRuled out
Look for Unauthorized / ErrImagePull events at the ECR egress boundary in ap-south-1.
The private ECR pull succeeds in 127ms with no auth errors. Registry and credentials are healthy.
No ErrImagePull or Unauthorized; pull completes in 127ms.
ECR access is fine; the registry is not the blocker.
Config assembly blocked by a missing Kubernetes SecretConfirmed
Inspect the kubelet config-assembly events and the container's envFrom specification to identify any missing references.
The kubelet cannot assemble the container environment: events show 'Error: secret "analytics-config" not found', repeating 71 times over 22 minutes. The container never runs. Confirmed; locate the Secret next.
Warning Failed with a secret-not-found message, recurring across the window.
LAST SEEN TYPE REASON MESSAGE
4m51s Warning Failed Error: secret "analytics-config" not found
count: 71 over 22m restarts: 0
Config assembly is blocked on an unresolved Secret reference, not on image or runtime.
The container pulls all env from a single Secret declared as a hard dependency.
[ { "secretRef": { "name": "analytics-config", "optional": false } } ]
Optional false makes the Secret mandatory; if it is missing the kubelet refuses to start the container.
Referenced Secret analytics-config is absent from the namespaceConfirmed
Query the prod namespace directly for analytics-config and list all secrets matching the name.
kubectl returns NotFound for analytics-config and the namespace-wide grep returns zero matches. The Secret does not exist in prod. The envFrom reference cannot resolve.
Direct get is NotFound; the listing has no analytics-config row.
$ kubectl get secret analytics-config -n prod
Error from server (NotFound): secrets "analytics-config" not found
$ kubectl get secrets -n prod | grep analytics
(no results)
The Secret is genuinely absent from the namespace, not in a wrong key or wrong case.
Secret exists but kubelet lacks RBAC permissions to read itRuled out
Check the kubelet's role bindings and cluster role for secret read permissions.
The kubelet has the correct permissions. The error is strictly "not found", not "forbidden".
Required Secret analytics-config missing from the prod namespace blocks config assemblyRoot cause
Tie the absent Secret to the recurring failure and the 0-restart waiting state, and confirm no transient delay explains it.
The mandatory Secret analytics-config is missing from prod, so the kubelet cannot assemble the container env and parks adhoc-worker in CreateContainerConfigError. 71 failures over 22 minutes with 0 restarts rule out a transient delay. Root cause. Restore the Secret and the kubelet retries automatically.
Steady-state failure, not a startup race.
error CreateContainerConfigError
cause secret "analytics-config" not found
failures 71
duration 22 min
restarts 0
A persistent, non-transient missing dependency. Creating the Secret unblocks config assembly without a manual restart.
ConfigMap referenced by envFrom is missingRuled out
Check the pod specification for any missing ConfigMap references.
The pod does not reference any ConfigMaps in its envFrom section, only the analytics-config Secret.
Final RCA
Required Secret analytics-config missing from the prod namespace blocks config assembly
The mandatory Secret analytics-config is missing from prod, so the kubelet cannot assemble the container env and parks adhoc-worker in CreateContainerConfigError. 71 failures over 22 minutes with 0 restarts rule out a transient delay. Root cause. Restore the Secret and the kubelet retries automatically.
Remediation
# Option 1: Recreate from known values
kubectl create secret generic analytics-config -n prod \
--from-env-file=.env.analytics-prod
# Option 2: Restore from backup / sealed-secret
kubectl apply -f sealed-secrets/prod/analytics-config.yaml
# Verify the pod recovers (no restart needed, the kubelet retries automatically)
kubectl get pod analytics-worker-0 -n prod -w- Determine how the secret was removed: check recent
kubectl deletecommands, Helm releases, or GitOps sync events that may have pruned it. - Store secrets in a sealed-secrets or external-secrets controller so they are declaratively managed and can be restored automatically.
- Add a pre-deployment check that validates all referenced secrets exist before rolling out new pods.
- Set up an alert for pods stuck in CreateContainerConfigError for more than 5 minutes to catch this faster.
A Secret referenced with optional false is a hard runtime dependency, so it should be owned by the same declarative pipeline that owns the Deployment rather than created out of band. Moving analytics-config into a sealed-secrets or external-secrets resource that lives beside the workload manifest means a GitOps sync recreates it instead of pruning it, and an accidental deletion self-heals on the next reconcile. Pair that with an admission check that rejects a rollout whose envFrom references do not resolve in the target namespace, and an alert on any container waiting in CreateContainerConfigError for more than five minutes. The first control stops the Secret from going missing, the second stops a pod from being scheduled against a Secret that already is, and the third bounds how long the gap can stay invisible.
Frequently asked questions
What causes CreateContainerConfigError in Kubernetes?
CreateContainerConfigError means the kubelet could not assemble the container's configuration before starting it, almost always because a referenced Secret or ConfigMap does not exist in the namespace. In this incident the container declared envFrom secretRef analytics-config with optional false, that Secret was absent from the prod namespace, and the kubelet refused to start the container. The image itself pulled successfully in 127ms.
What is the difference between CreateContainerConfigError and CrashLoopBackOff?
CreateContainerConfigError happens before the container process runs, so the pod sits in Waiting with a restart count of zero. CrashLoopBackOff happens after the process has started and exited, so the state is Terminated and the restart count climbs. Checking restart count settles it immediately: zero restarts over 22 minutes means the container never executed and the fault is in configuration, not in application code.
Does a Kubernetes pod recover on its own after you create the missing Secret?
Yes. The kubelet retries container creation on its normal sync loop, so creating the Secret in the correct namespace unblocks config assembly without deleting the pod or restarting the Deployment. In this incident the kubelet had already retried 71 times over 22 minutes. Watch the pod with kubectl get pod -w and it moves out of Waiting once the reference resolves.
Should envFrom secretRef be marked optional to avoid this failure?
No. optional false is what made the failure loud and immediate rather than silent. With optional true the kubelet starts the container with the environment variables simply absent, and the workload fails later with missing credentials or falls back to defaults, which is far harder to diagnose. Keep the hard dependency and fix the missing Secret instead.