Alert info
- Severity
- Warning
- Detected by
- CloudWatch
- Alert
- CurrConnections exceeded 1,500 threshold on eaze-websocket-001
- Time
- 2026-04-17 07:59:20 UTC
- Service
- eaze-websocket-001 (Redis · production)
CurrConnections on eaze-websocket-001 jumped from ~280 to 1,576.6 avg / 1,788 max in a single 5-minute bucket at 07:54Z. Investigate the cause.
HPA-driven pod churn on eaze-channels pushed CurrConnections from ~280 to 1,788 on Redis for one 5-minute bucket. Cache health stayed normal; transient connection fan-out from scaling, not a deploy or resource saturation.
- CurrConnections
- 1,788 max
- NewConnections
- 5,762
- Engine CPU
- 4-5%
Entities identified
Infrastructure
eaze-channels HPA
autoscaler
HorizontalPodAutoscaler managing eaze-channels deployment.
new pods
2 added on scale-up
Pods backing the eaze-channels deployment.
eaze-channels rev 224
deploy 2026-04-15
Latest deployment revision for eaze-channels.
Service
eaze-channels
websocket workload
HPA-managed Deployment serving websocket channels.
eaze-celery-websockets
worker
Second listed caller of
eaze-websocket-001. No per-client connection breakdown available; its share of the burst is unquantified.
Data
eaze-websocket-001
Redis · production
ElastiCache Redis instance in
ap-south-1handling websocket connections.
Gateway
CloudWatch alarm
CurrConnections > 1,500
CloudWatch alarm tracking CurrConnections on ElastiCache.
How they relate
The same relationships as text
- eaze-channels HPA scales eaze-channels. HPA scales the deployment.
- eaze-channels spawns new pods. Deployment creates pods.
- new pods opens connections eaze-websocket-001. Pods connect to Redis.
- eaze-channels websocket fan-out eaze-websocket-001. Steady-state eaze-channels connections to Redis; surged transiently as pods churned.
- eaze-celery-websockets connects eaze-websocket-001. Second caller of
eaze-websocket-001. Contribution to the burst not quantifiable from available telemetry. - eaze-channels rev 224 last shipped eaze-channels. Revision 224 landed 2 days prior. No deploy in the incident window.
- eaze-websocket-001 triggers CloudWatch alarm. Alarm monitors Redis connection counts.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
A new eaze-channels deploy regressed connection handlingRuled out
Diff eaze-channels rollout history against the 07:54Z spike to see if a recent deploy caused this.
No new revision landed at or near the incident. Latest image is 2 days old. A deploy-triggered burst is ruled out by rollout history.
Latest revision and image age for eaze-channels.
REVISION IMAGE CREATED
224 prod/eaze:2026-04-15-18-29-2 2d ago latest
no new revision at or near 07:54Z 2026-04-17
No deploy in the incident window. No temporal link to the spike.
Redis resource saturation (CPU or memory) backed up connectionsRuled out
Pull EngineCPUUtilization and DatabaseMemoryUsagePercentage across the 07:54Z window.
EngineCPU 4-5% and memory 2.4-2.5% throughout. The cache was near idle on both axes. No saturation, no eviction backlog.
Engine CPU and memory utilization during the incident window.
metric value status
EngineCPUUtilization 4-5% ok
DatabaseMemoryUsagePercent 2.4-2.5% ok
CPU [#-------------------] 5%
Memory [#-------------------] 3%
Backend is idle. This is a connection-count event, not Redis pressure. Ruled out.
Gradual organic load growth crossed the thresholdRuled out
Compare the spike shape against same-hour baselines 1 day and 7 days prior.
CurrConnections stepped from 280.6 to 1,576.6 in a single 5-minute bucket. Same-hour history was 257-535 (1d) and 533-796 (7d), well under 1,500. A one-bucket step is inconsistent with gradual growth.
Same-hour CurrConnections on prior days vs the incident bucket.
date CurrConnections avg
2026-04-10 (7d ago) 533 - 796
2026-04-16 (1d ago) 257 - 535
2026-04-17 (incident) 1,576.6 at 07:54Z
2-3x historical peak in one bucket. Not organic growth. Ruled out.
Upstream pod churn drove a transient Redis connection fan-outConfirmed
Read NewConnections alongside CurrConnections to check for a connection surge.
CurrConnections and NewConnections spike in the same bucket: active connection creation, not gradual accumulation. The fingerprint of a client/pod reconnect burst. Confirmed; trace which client churned.
Both metrics jump in the same 5-minute bucket.
timestamp CurrConn avg CurrConn max NewConn sum
07:44:00Z ~300 - 4,260
07:49:00Z 280.6 - 4,728
07:54:00Z 1,576.6 1,788 5,762 breach
Simultaneous creation and accumulation is a reconnect burst, not load growth. Cause is a churning client.
The churning client is HPA-scaled eaze-channelsConfirmed
Query HPA status and scaling events for eaze-channels to determine if scaling occurred.
eaze-channels oscillated 12 to 14 to 12 in the minutes before 07:54Z with 1 pod still unavailable. Scale events line up with the connection burst in time and mechanism.
Recent scale-up/down activity on eaze-channels.
NAME TARGETS MINPODS MAXPODS REPLICAS
eaze-channels 39%/60% 2 15 12
~34m ago scaled up 12 -> 14
~24m ago scaled down 14 -> 13
~23m ago scaled down 13 -> 12
current: 12 desired, 11 available, 1 unavailable
HPA churn is the upstream trigger; its timing matches the burst.
New pods each open an uncapped Redis client poolRoot cause
Correlate scale-up events with the NewConnections metric to see if new pods are creating excessive connections.
The 12 to 14 scale-up added 2 pods, each initializing a fresh Redis client pool against eaze-websocket-001 with no observed max-connection cap. That fan-out drove NewConnections to 5,762 and CurrConnections to 1,788 for one bucket. Root cause. Transient: recovering by 07:56Z (1,485), alarm OK at 08:01:15Z.
Connection volume vs the 2 pods added on scale-up.
scale-up 12 -> 14 (+2 pods)
NewConn sum 4,728 -> 5,762 (+22%)
CurrConn max 280 -> 1,788
1,788 connections from only ~2 extra pods implies a high, uncapped per-pod pool. Root cause.
Breach duration after the burst.
07:54:00Z 1,576.6 avg / 1,788 max breach
07:56:00Z 1,485.0 recovering
08:01:15Z alarm OK
One 5-minute evaluation period. A one-time pod-churn burst, not a sustained fault.
Exact per-pod pool size and eaze-celery-websockets shareNeeds access
Attempt to retrieve the Redis client environment variables and a per-client connection breakdown.
The precise per-pod pool cap and the split between eaze-channels and eaze-celery-websockets are outside the available telemetry.
No per-client connection breakdown and no Redis client pool config in the available data.
We can name the mechanism but not the exact pool size or the second caller's share. Needs privileged access.
Redis timeout configuration causes connection accumulationRuled out
Review Redis timeout and keepalive configurations.
Timeouts are configured appropriately, the issue was purely the burst of new connections from large pools.
HPA scaling was driven by a memory leakRuled out
Check the pod memory utilization to see if it drove scaling.
HPA is scaled on CPU, and memory was stable.
Celery workers scaled up and opened many connectionsRuled out
Check the scaling events and replica count for eaze-celery-websockets.
eaze-celery-websockets replica count remained stable during the incident window.
Final RCA
New pods each open an uncapped Redis client pool
The 12 to 14 scale-up added 2 pods, each initializing a fresh Redis client pool against eaze-websocket-001 with no observed max-connection cap. That fan-out drove NewConnections to 5,762 and CurrConnections to 1,788 for one bucket. Root cause. Transient: recovering by 07:56Z (1,485), alarm OK at 08:01:15Z.
Remediation
# Check current connection pool settings in eaze-channels Redis client config
kubectl exec -n prod deploy/eaze-channels -- env | grep -i 'redis\|conn\|pool'
# Review HPA behavior and consider smoothing scale-down
kubectl get hpa eaze-channels -n prod -o yaml | grep -A10 behavior
# Add NewConnections rate-of-change alarm for earlier detection
aws cloudwatch put-metric-alarm --alarm-name eaze-websocket-001-NewConnections-RoC \
--metric-name NewConnections --namespace AWS/ElastiCache \
--statistic Sum --period 300 --threshold 6000 --comparison-operator GreaterThanThreshold- Implement connection pooling with a max-connections cap in upstream Redis clients to bound the connection fan-out when pods churn.
- Configure HPA scale-down stabilization window (e.g., 300s) to reduce rapid replica oscillations that trigger reconnect bursts.
- Add a CloudWatch alarm on NewConnections rate-of-change to detect reconnect bursts before CurrConnections breaches the threshold.
- Tune the CurrConnections alarm with a longer evaluation period (e.g., 2 of 3 datapoints) to avoid firing on single-bucket transient spikes.
- Audit connection lifecycle in
eaze-celery-websocketsto quantify its contribution to Redis connection load during scaling events.
A pod-churn connection burst like this is a symptom of two independent gaps that happen to line up in time: the Redis client has no max-connections cap, so each new pod opens as many connections as it wants, and the HPA has no scale-down stabilization window, so replica counts oscillate more than the workload requires. Capping the client pool bounds the fan-out per pod regardless of how many pods start at once, and a stabilization window on the HPA reduces how often that fan-out is triggered in the first place. Layering a rate-of-change alarm on NewConnections, rather than only a static threshold on CurrConnections, catches the next burst while it is still building instead of after it has already breached.
Frequently asked questions
Why do HPA scale-up events cause a Redis connection spike?
Each new pod created during an HPA scale-up initializes its own Redis client connection pool from scratch. If that pool has no max-connections cap, a small number of new pods can open a disproportionately large number of connections at once. In this incident, eaze-channels scaled from 12 to 14 replicas and NewConnections jumped from about 4,700 to 5,762 in the same 5-minute bucket, driving CurrConnections to 1,788.
How do you tell a connection burst apart from Redis resource saturation?
Check EngineCPUUtilization and DatabaseMemoryUsagePercentage during the spike. In this incident both stayed low (4-5% CPU, 2.4-2.5% memory) throughout the breach, ruling out CPU or memory pressure as the cause. When CurrConnections and NewConnections spike together while cache health metrics remain flat, the event is client-driven connection creation, not backend saturation.
Was this ElastiCache connection breach caused by a bad deploy?
No. Rollout history showed the latest eaze-channels revision was two days old with no new deployment at or near the incident time. The connection spike instead correlated with HPA scaling events (12 to 14 to 12 replicas) in the minutes before the breach, which pointed to pod churn rather than a code or configuration regression.
How do you prevent repeat CurrConnections alarms from routine autoscaling?
Cap the Redis client connection pool per pod so scale-up events cannot fan out into thousands of new connections, and add an HPA scale-down stabilization window to reduce oscillation. Pairing a NewConnections rate-of-change alarm with a longer evaluation period on the CurrConnections alarm catches genuine reconnect bursts earlier while avoiding false alarms from a single transient bucket.