How do you trace an Aurora PostgreSQL load surge from a downstream symptom?

The alert says a Celery worker is at 91 percent of its memory limit. The worker did not change, its traffic did not change, and its memory has never gone above 60 percent before. Somewhere else, a database got slower, and everything in flight started waiting.

Why does a database surge show up somewhere else first?#

Because the application holds the work while the database is busy.

A worker that processes a task in 200 ms holds one task's worth of memory. If the query behind that task starts taking 2 seconds, the worker holds ten times as much at the same arrival rate, because concurrency equals arrival rate multiplied by hold time. Nothing about the worker changed. Its hold time did, and hold time is set by the database.

The same arithmetic explains the queue. Arrivals are unchanged, service time is up, so the backlog grows and keeps growing until service time recovers. By the time anyone is paged, the visible symptom is memory or queue depth in a service that is entirely innocent.

This is why resizing the worker so often fails. A bigger memory limit lets it hold a longer backlog. It does not make the query faster, and the backlog keeps growing to fill whatever you give it.

How do you confirm the database is the cause?#

Look for the surge and the symptom in the same window, then read what the database was waiting on.

# Aurora's own view of load, in average active sessions. This is the# single best "is the database busy" metric, and it is comparable to vCPU count.aws cloudwatch get-metric-statistics \  --namespace AWS/RDS --metric-name DBLoad \  --dimensions Name=DBClusterIdentifier,Value=moderation-prod \  --statistics Average --period 60 \  --start-time 2026-04-17T06:00:00Z --end-time 2026-04-17T06:40:00Z \  --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Average]' --output text 

DBLoad above the cluster's vCPU count means sessions are queueing for CPU. Then ask the database what they were queueing on:

-- What is actually running, grouped by what it is waiting for.-- wait_event_type is the diagnosis: CPU, IO, Lock and LWLock are-- four different incidents with four different fixes.SELECT count(*) AS sessions,       state,       wait_event_type,       wait_eventFROM pg_stat_activityWHERE backend_type = 'client backend'GROUP BY 2, 3, 4ORDER BY 1 DESC; 

 sessions | state  | wait_event_type | wait_event----------+--------+-----------------+---------------      184 | active | IO              | DataFileRead       31 | active | LWLock          | BufferMapping        9 | active | CPU             |        2 | idle in transaction |    | 

IO / DataFileRead at that volume means queries are reading from storage rather than the buffer cache, which is what a plan regression or a missing index looks like from the outside. idle in transaction sessions are worth noting separately: they hold locks and connections without doing work.

Then find the statements responsible:

-- Top statements by total time. Requires pg_stat_statements, which-- Aurora enables through shared_preload_libraries.SELECT calls,       round(mean_exec_time::numeric, 1) AS mean_ms,       round(total_exec_time::numeric / 1000, 1) AS total_s,       rows,       left(query, 90) AS queryFROM pg_stat_statementsORDER BY total_exec_time DESCLIMIT 10; 

Mean execution time is the number to compare against yesterday. A statement that moved from 4 ms to 900 ms at unchanged call volume is your surge, and it multiplies held connections by 225 without a single extra request arriving.

How do you confirm the downstream symptom is a consequence?#

Show that the worker's memory tracks the database's latency rather than its own traffic.

# The worker side. Memory as a fraction of limit, and the queue behind it.kubectl top pod -n prod -l app=moderation-chat-celery --containers kubectl get pods -n prod -l app=moderation-chat-celery \  -o custom-columns='NAME:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[0].restartCount' # And why a pod left, which is not always what you assume.kubectl get events -n prod --sort-by=.lastTimestamp \  --field-selector involvedObject.kind=Pod | grep -iE 'evict|kill|preempt' | tail 

The distinction that matters here is eviction versus OOMKill. An Evicted pod with reason Underutilized was removed by a descheduler or autoscaler, which is a completely different event from the kernel killing a container for exceeding its limit. Confusing the two sends the investigation to the wrong place, and the difference is covered in the Kubernetes OOMKilled guide.

How do you fix an Aurora load surge?#

Fix the query, then give the worker enough headroom to survive the next one.

If a plan regressed, that is the whole incident. Confirm it with EXPLAIN (ANALYZE, BUFFERS) and look for a sequential scan where an index scan used to be, then add or restore the index. A missing index changes the cost of one statement by two orders of magnitude, and no amount of worker memory compensates.

If reads are saturating the writer, move them. Aurora replicas exist for this, and the reader endpoint costs nothing to adopt for queries that tolerate replica lag:

aws rds describe-db-clusters --db-cluster-identifier moderation-prod \  --query 'DBClusters[].{Writer:Endpoint,Reader:ReaderEndpoint,Capacity:ServerlessV2ScalingConfiguration}' \  --output table 

If the surge is legitimate load, bound its blast radius. A statement timeout stops one pathological query from holding a connection indefinitely, and it turns an unbounded incident into a bounded one:

-- Per-role, so a batch role can be generous and the web role cannot be.ALTER ROLE app_web SET statement_timeout = '5s';ALTER ROLE app_web SET idle_in_transaction_session_timeout = '10s'; 

On the worker side, cap the prefetch so a slow database cannot fill memory with work in flight. This is the change that actually protects the pod, because it makes hold time stop translating directly into held memory:

env:  # One task in flight per process. Without this, a worker prefetches a  # batch and holds all of it while the database is slow.  - name: CELERYD_PREFETCH_MULTIPLIER    value: "1"  - name: CELERYD_MAX_TASKS_PER_CHILD    value: "100"resources:  requests:    memory: 768Mi  limits:    memory: 1Gi 

What does a real Aurora load surge look like?#

Sherlocks AI investigated one that arrived as a Kubernetes memory alert. A moderation-chat-celery worker in prod breached 90 percent of its 1Gi limit, measured at 91 percent, sustained for five minutes at 06:22 UTC.

The worker was not leaking. Its task backlog was growing during a concurrent Aurora PostgreSQL load surge, and the backlog was what occupied memory. The pod that left was evicted for Underutilized, not OOMKilled, and the surviving replica still sat at 87 percent. The finding was thin headroom against a database-driven backlog, not a crash and not a leak in the worker. The full writeup is in the Celery memory pressure investigation.

Two things make this pattern hard. The paging service is never the responsible one, and the two plausible-looking causes, a leak and an OOMKill, are both wrong in a way that sends you to the worker's code instead of the database's plans. More AWS incidents worked end to end are on the AWS examples page, the connection-side version of the same story is in the ElastiCache connection spikes guide, and the engine's own reference for wait events is the PostgreSQL monitoring documentation alongside Amazon RDS Performance Insights.

Aurora load surge FAQ#

Why did a database surge page me about a worker's memory? Because the worker holds work while the database is slow. Concurrency equals arrival rate times hold time, and the database sets hold time.

Will giving the worker more memory fix it? No. It lets the worker hold a longer backlog. The backlog grows to fill whatever you give it until query time recovers.

What is DBLoad and what value is bad? Average active sessions. Sustained above the cluster's vCPU count means sessions are queueing rather than running.

How do I tell an eviction from an OOMKill? An eviction is the scheduler removing a pod and carries a reason like Underutilized. An OOMKill is the kernel, and shows Reason: OOMKilled with exit code 137.

What usually causes the surge itself? A plan regression after a migration, a missing index, or read traffic saturating the writer. Compare mean execution time per statement against a normal day.

Does adding an Aurora replica help? Only for queries that can tolerate replica lag. It relieves the writer, it does not repair a regressed plan.