Alert info
- Severity
- Warning
- Detected by
- Grafana
- Alert
- Avg memory usage > 90% of limit for 5m on moderation-chat-celery
- Time
- 2026-04-17 06:22:24 UTC
- Service
- moderation-chat-celery (prod)
Average memory usage breached 90% of the 1Gi limit (measured 91%) for 5 minutes. By investigation time the alerted pod was gone, replaced by a fresh replica, while a surviving replica still sat at 87%.
Celery worker hit 91% of 1Gi memory limit as task backlog grew during a concurrent Aurora PostgreSQL load surge. Pod was evicted for Underutilized (not OOMKilled); surviving replica still at 87%. Thin headroom, not a crash.
- Memory % Limit
- 91%
- Surviving Pod
- 891Mi (87%)
- DB Connections
- ~75 → 124
Entities identified
Infrastructure
broker / queue
task backlog
Celery broker holding the moderation task queue. Backlog grew as task completion slowed against the DB surge.
pod ...-ftmgt
evicted
Alerted pod
moderation-chat-celery-844d566f58-ftmgt. Replaced by a fresh replica.pod ...-zz5tj
surviving replica
Surviving 120m-old replica.
moderation rev 79
deploy 2026-04-16
Latest rollout (rev 79) image moderation:2026-04-16-19-46-282, ~11h before the alert. No revision created at 06:22Z. Rules out a deploy trigger.
Service
moderation-chat-celery
K8s deploy (prod)
Celery worker. Alerted pod hit 91% of 1Gi limit at 06:22Z. Limits 1Gi / requests 768Mi. No 5xx, no crash signatures.
Data
moderation-writer
Aurora PostgreSQL
Downstream Aurora PostgreSQL database instance used for moderation writes.
External
ML/audio libraries
worker startup
ML/audio libraries loaded at worker startup (repeated WARN/INFO). Inflates baseline memory, contributing to the tight 87% steady state but not the timing.
Gateway
lokalapp-prod
caller
Upstream app that enqueues moderation work onto the celery broker. Traffic itself normal in the window.
How they relate
The same relationships as text
- lokalapp-prod enqueues tasks broker / queue
- broker / queue dispatches moderation-chat-celery. Workers pull moderation tasks. Backlog held in-memory task state as completion slowed.
- moderation-chat-celery queries moderation-writer. Workers make per-task DB calls to the moderation-writer.
- moderation-writer responses moderation-chat-celery. Aurora responds to moderation-writer queries.
- moderation-chat-celery alerted pod pod ...-ftmgt
- moderation-chat-celery replica pod ...-zz5tj
- moderation rev 79 last rollout moderation-chat-celery
- ML/audio libraries loaded at startup moderation-chat-celery
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Pod was OOMKilled by the kernel out-of-memory killerRuled out
Read Kubernetes events for the alerted pod to confirm the termination reason and check application logs for OOM/MemoryError traces.
Events show the pod was Evicted with reason Underutilized (node consolidation), not OOMKilled. Replacement started cleanly with zero restarts; no MemoryError in logs.
The alerted pod was evicted for Underutilized, then a replacement was scheduled and started.
LAST SEEN TYPE REASON MESSAGE
~18m Normal Evicted Evicted pod: Underutilized
~18m Normal Killing Stopping container moderation-chat-celery
~17m Normal Scheduled pod/...-x55tb assigned to node
~17m Normal Started Started container moderation-chat-celery
Eviction for node consolidation, not an OOMKilled kernel termination. Not a fatal memory breach.
No MemoryError or OOMKilled traces across the incident window.
pattern count level
MemoryError / OOMKilled 0 ERROR
No hard crash. Consistent with gradual backlog-driven retention, not a kill.
A recent deploy introduced a leaking code pathRuled out
Diff rollout history against the 06:22Z alert to see if a fresh revision was deployed.
Latest rollout (rev 79) image is from 2026-04-16-19-46-282, ~11h before the alert. No revision was created around 06:22Z on 2026-04-17.
No new revision near the alert time; rev 79 had been live ~11h.
REVISION IMAGE CREATED
79 moderation:2026-04-16-19-46-282 ~11h ago <- latest
78 moderation:2026-04-16-17-23-281 ~13h ago
No deploy in the window. A code-regression leak would need a fresh revision. Ruled out.
Transient spike / monitoring artifact, not sustained pressureRuled out
Check whether the condition persisted by reading live usage on the surviving replica long after the alert.
The alert rule requires avg memory > 90% sustained for 5m. The surviving replica was still at 891Mi (87%) two hours later. The condition was real and persistent, not a blip.
The 120m-old surviving pod was still near the limit hours after the alert.
NAME CPU MEMORY
...-x55tb 18m 694Mi <- 6m old (replacement)
...-zz5tj 24m 891Mi <- 120m old (surviving)
Limit [####################] 91% (value at alert)
Live [##################--] 87% (891Mi / 1Gi, surviving)
Pressure is workload-wide and sustained, not a transient artifact. Ruled out.
Task backlog held memory near the fixed 1Gi limit during a load surgeConfirmed
Inspect the resource spec and live usage for headroom, then look downstream for what could be slowing task completion in the alert window.
Limits are 1Gi / requests 768Mi; steady-state usage is already 87% (891Mi), leaving only ~130Mi before the 90% threshold. Workers retain in-memory task state, so anything that slows completion breaches fast. Confirmed; trace what slowed completion.
Steady-state usage sits within ~130Mi of the 90% alert threshold.
requests.memory 768Mi
limits.memory 1Gi
surviving pod 891Mi (87%)
headroom to 90% ~130Mi
Thin headroom. The effect of any backlog is immediately visible at the limit.
Repeated celery missed-heartbeat entries during the window, with no crash traces.
pattern count level
celery missed heartbeat many WARN
ML/audio library loading repeated WARN/INFO
Workers are stalling, not crashing. Task state is held, not released. Cause is downstream.
Redis broker degraded performance slowed task ingestionRuled out
Query Redis broker metrics (CPU, memory, evictions) to see if the broker itself was slow to dispatch.
Broker metrics were healthy. The backlog formed because workers held tasks, not because the broker couldn't deliver them.
Aurora PostgreSQL load surge slowed task completion in the same windowConfirmed
Query Aurora moderation-writer CPU, connections, active sessions, and latency for 06:00Z-06:30Z and correlate with the memory alert.
Aurora surged 06:10-06:25Z: connections ~75 to ~124, active sessions ~6-8 to ~19-21, CPU ~30% to ~46%, while latency stayed low. Busier but not saturated, so tasks slowed and queued rather than erroring.
Connections, sessions, and CPU rose sharply in the same window as the alert; latency stayed low.
metric baseline 06:00Z peak 06:15-06:25Z
CPU utilization ~30-33% ~46% at 06:20
connections ~75-78 124 at 06:25
active sessions ~5-8 ~19 at 06:20 / 20.6 at 06:25
read latency ~1.1-2.0ms slight increase
DB busier but not latency-saturated. Higher concurrency slowed individual task completion.
The DB surge window and the memory alert window line up.
06:10Z Aurora connections begin climbing
06:20Z active sessions ~3x baseline, CPU ~46%
06:22Z memory alert fires at 91% of 1Gi
Slower completion keeps workers occupied, retaining task state against thin headroom.
Aurora DB connections exhausted causing application crashesRuled out
Check application logs for DB connection timeout or exhaustion errors.
No connection errors were logged. The DB had higher concurrency but continued to serve queries.
Backlog-driven in-memory task retention against the fixed limitRoot cause
Combine findings from DB load and worker memory limits to verify if slowed DB completion caused celery prefetch to hold task payloads against the memory limit.
moderation-chat-celery hit 91% because work backed up during the Aurora surge: workers held more in-memory task state while completion slowed, against a 1Gi limit already 87% full. No OOM, no crash; the pod was evicted for Underutilized and service stayed up via the replacement.
Slow DB completion + worker prefetch + thin headroom produce the 91% breach without a crash.
DB sessions ~6-8 -> ~19-21 (slower completion)
task state held in worker memory while awaiting DB
limit 1Gi, steady 87% (891Mi), ~130Mi headroom
result 91% breach, evicted Underutilized (not OOM)
Load-driven memory pressure against a tight limit. The fix is headroom + prefetch + DB pooling.
Exact upstream trigger for the DB surge + celery prefetch configNeeds access
Attempt to check the upstream job scheduler and the worker's prefetch/concurrency environment to determine the exact trigger.
Not in the captured telemetry. The specific batch/backfill job that drove the 06:10-06:25Z surge and the exact prefetch multiplier need privileged access to confirm.
Prometheus historical memory series returned empty; the upstream surge trigger was not identified.
The investigation labels the boundary rather than guessing. Needs access to the job scheduler and worker config.
Malicious payload or specific large message caused the memory spikeRuled out
Inspect message sizes and payload formats for anomalies.
Payload sizes were typical. The issue was the volume of concurrently held tasks, not a single massive task.
Final RCA
Backlog-driven in-memory task retention against the fixed limit
moderation-chat-celery hit 91% because work backed up during the Aurora surge: workers held more in-memory task state while completion slowed, against a 1Gi limit already 87% full. No OOM, no crash; the pod was evicted for Underutilized and service stayed up via the replacement.
Remediation
# Increase memory limit to provide more headroom
kubectl set resources deploy/moderation-chat-celery -n prod -c celery \
--limits=memory=1536Mi --requests=memory=1024Mi
# Check celery prefetch and concurrency settings
kubectl exec -n prod deploy/moderation-chat-celery -- env | grep -iE 'prefetch|concurrency|pool'
# Monitor DB connection pressure
aws rds describe-db-instances --db-instance-identifier moderation-writer \
--query 'DBInstances[0].Endpoint'- Increase memory limit from 1Gi to 1.5Gi (and request from 768Mi to 1Gi) to provide headroom for task backlog during load surges.
- Reduce celery worker prefetch multiplier (e.g., from 4 to 1) so workers only fetch tasks they can process immediately, reducing in-memory task state.
- Add connection pooling (PgBouncer or equivalent) between celery workers and Aurora to bound concurrent DB sessions and prevent task pileup.
- Investigate the upstream trigger for the DB load surge at 06:10-06:25Z: bulk job, batch client, or config change.
- Add a Prometheus alert on Aurora active sessions exceeding baseline by 2x, providing earlier warning before downstream worker memory pressure builds.
The alert fired at 91% not because anything crashed, but because a fixed 1Gi memory limit left only about 130Mi of headroom above an already tight 87% steady state. Raising the limit buys back headroom, but the more durable fix is to stop workers from holding task state longer than they need to: a lower prefetch multiplier limits how many tasks a worker can pull ahead of what it can execute, and connection pooling in front of Aurora bounds how many concurrent sessions a downstream slowdown can create. Together those changes mean a future DB load surge slows task throughput without also inflating worker memory, and a session-count alert on Aurora gives warning before the celery-side symptom appears at all.
Frequently asked questions
Why did a Celery worker's memory usage spike without an OOMKill?
Kubernetes events showed the alerted pod was evicted with reason "Underutilized", a node consolidation action, not the kernel's out-of-memory killer. The replacement pod started cleanly with zero restarts, and application logs had no MemoryError or OOMKilled traces. The 91% reading reflected sustained backlog pressure against a tight limit, not a fatal memory breach.
How does a downstream database load surge cause a worker to run out of memory headroom?
When Aurora PostgreSQL gets busier, connections and active sessions rise even if the database is not saturated. In this incident, active sessions roughly tripled from about 6-8 to 19-21 while read latency stayed low. Slower per-task completion meant celery workers held more in-memory task state at once, and with only about 130Mi of headroom above the 87% steady state, that backlog pushed usage to 91%.
Was this Celery memory alert caused by a code change or memory leak?
No. Rollout history showed the latest revision was about 11 hours old with no deployment near the alert time, ruling out a fresh code-regression leak. Logs showed repeated missed-heartbeat entries during the window, consistent with workers stalling on slower downstream responses rather than crashing or leaking memory.
What is the fix for thin memory headroom on a Celery worker?
Increase the memory limit and request to restore headroom, but also address why headroom disappears under load: reduce the worker prefetch multiplier so it only holds as many tasks in memory as it can immediately process, and add connection pooling in front of the database so a session surge does not translate directly into worker backlog. An alert on Aurora active sessions exceeding baseline gives earlier warning than waiting for the worker-side memory alert.