Alert info
- Severity
- Warning
- Detected by
- CloudWatch
- Alert
- CurrConnections on eaze-websocket-001 exceeded threshold
- Time
- 2026-04-06 07:17:33 UTC
- Service
- eaze-websocket-001 (Redis · production)
CurrConnections on the Redis layer jumped from a ~370 baseline to 1,988 max in a single 5-minute window at 07:12Z. Investigate the cause.
WebSocket reconnect/reject loop on eaze-channels drove CurrConnections from ~370 to 1,988 on Redis. Cache health stayed normal; root cause was WSREJECT on /ws/expert_availability causing client-side churn.
- CurrConnections
- 1,988 max
- NewConnections
- 6,377
- Redis CPU
- 6.6%
Entities identified
Infrastructure
eaze-channels rollout
deploy (unknown)
Rollout history for eaze-channels and
eaze-celery-websockets.
Service
eaze-channels
Django Channels
Owns the WS route.
eaze-celery-websockets
worker
Upstream caller in the infra graph, shares the Redis layer.
Data
eaze-websocket-001
Redis · production
ElastiCache Redis instance.
External
WS clients
browser / app
WebSocket clients subscribing to expert availability.
Gateway
/ws/expert_availability
WS route
Daphne/Channels WebSocket route.
How they relate
The same relationships as text
- WS clients connect / upgrade /ws/expert_availability. Clients open WebSocket upgrades against the route.
- /ws/expert_availability WSREJECT WS clients. Route responds to the upgrade.
- /ws/expert_availability served by eaze-channels
- eaze-channels opens connection eaze-websocket-001. Service opens Redis connections.
- eaze-celery-websockets shares layer eaze-websocket-001
- eaze-channels rollout shipped (?) eaze-channels
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Redis-side resource exhaustion (CPU, memory, or maxclients)Ruled out
Query eaze-websocket-001 CPUUtilization, EngineCPUUtilization, and DatabaseMemoryUsagePercentage across the window.
CPU 6.57% (max 9.26%), EngineCPU 5.03%, DB memory 2.34% (max 2.44%). All well within safe limits the whole window. Redis is a passive victim, not the cause.
Resource utilization stayed low throughout the spike.
metric avg max status
CPUUtilization 6.57% 9.26% ok
EngineCPUUtilization 5.03% 5.30% ok
DatabaseMemoryUsagePercentage 2.34% 2.44% ok
No saturation, no memory pressure. If maxclients were hit, CurrConnections would plateau at the limit; it peaked at 1,988 and fell. Ruled out.
Slow accumulation of idle / leaked connectionsRuled out
Compare CurrConnections rise against NewConnections rate for the window.
NewConnections surged to 6,377 in the same 5-minute window as the CurrConnections jump. A leak shows gradual CurrConnections growth with a low NewConnections rate; the simultaneous spike in both is a churn loop, not idle accumulation.
Both metrics spiked together rather than diverging.
window 07:12Z
CurrConnections ~370 -> 1,666 avg / 1,988 max
NewConnections 6,377 sum (1,275/sec avg)
Rapid create/teardown, the fingerprint of a reconnect loop, not a slow leak. Ruled out.
Recent deploy to eaze-channels introduced the regressionRuled out
Check rollout history for eaze-channels and eaze-celery-websockets to identify recent deployments.
Rollout history could not be confirmed from available data; no image change or deploy correlation could be proven in the window. Cannot be used as the cause. Ruled out here; tracked as an open caveat.
Deployment timestamps for both services were not retrievable from available data.
No deploy-to-onset link could be established. A deploy may have initiated WSREJECT but it is unverified, so it is not the proven cause.
Client-driven connection storm from WebSocket reconnect churnConfirmed
Check eaze-channels application logs for connection frequency and correlate with the Redis connection spike.
eaze-channels logs show high-frequency connect/disconnect cycles aligned with the 07:12Z spike. CurrConnections rose 4.5x and NewConnections hit 6,377 in one interval: a client-driven storm, not a Redis problem. Confirmed; trace the trigger for the churn next.
Dense WSCONNECTING/WSCONNECT/WSDISCONNECT cycles in the spike window.
eaze-channels logs (06:17Z - 07:17Z)
pattern count level
WSCONNECTING / WSCONNECT / WSDISCONNECT high INFO
Rapid create/teardown at the application layer is what opens new Redis connections. The storm originates from the WS clients, not the cache.
WSREJECT on /ws/expert_availability triggered a reconnect loopRoot cause
Search eaze-channels logs for WSREJECT and verify if clients are reconnecting.
Repeated WSREJECT on /ws/expert_availability interleaves with the connect/disconnect churn. Each reject makes the client re-attempt with no backoff, opening a fresh Redis connection per attempt: ~21 reconnects/sec drove CurrConnections to 1,988 and NewConnections to 6,377 while Redis stayed healthy.
WSREJECT recurs specifically on /ws/expert_availability.
[11:54:38] INFO "WSREJECT /ws/expert_availability"
WSREJECT /ws/expert_availability repeated INFO
The route is rejecting the WS upgrade. The reject is the trigger that feeds the reconnect loop.
New connection rate implied by the 5-minute NewConnections sum.
NewConnections 6,377 / 5 min
implied rate ~21 reconnects/sec
Clients retry rejected upgrades immediately with no backoff, amplifying each reject into a Redis connection storm.
Exact WSREJECT trigger (auth / CORS / Daphne route config)Needs access
Attempt to retrieve verbose Daphne/Channels access logs and configuration diffs to determine the reject reason.
The reject reason is not present in available logs. Candidate triggers are an auth/token validation change, a CORS/allowed-hosts change, or a Daphne consumer-route change, but none could be confirmed from available data.
Available Daphne/Channels logs record WSREJECT but not why the upgrade was rejected.
The investigation labels the reject as the trigger and stops at the telemetry boundary rather than guessing the exact cause.
Client-side bug forces reconnects regardless of route statusRuled out
Check client application logs and bug reports.
Clients behave correctly by retrying; the server-side WSREJECT is the root issue driving the loop.
Load balancer health checks are creating transient connectionsRuled out
Check load balancer health check logs to see if they correlate with the surge.
Health checks run at a steady low frequency and do not account for the surge.
Final RCA
WSREJECT on /ws/expert_availability triggered a reconnect loop
Repeated WSREJECT on /ws/expert_availability interleaves with the connect/disconnect churn. Each reject makes the client re-attempt with no backoff, opening a fresh Redis connection per attempt: ~21 reconnects/sec drove CurrConnections to 1,988 and NewConnections to 6,377 while Redis stayed healthy.
Remediation
# Check recent changes to websocket auth/token validation
git log --since='2026-04-05' --oneline -- '**/ws/**' '**/expert_availability**'
# Review Daphne/Channels WSREJECT reason in verbose logs
kubectl logs -n prod -l app=eaze-channels --since=2h | grep -i 'WSREJECT\|expert_availability'
# Check CORS / allowed-hosts configuration
kubectl get configmap eaze-channels-config -n prod -o yaml | grep -i 'origin\|cors\|allowed'- Investigate the root cause of WSREJECT on
/ws/expert_availability: check recent auth, CORS, or allowed-host changes. - Add connection rate limiting or backoff logic to websocket clients to prevent reconnect storms from cascading to Redis.
- Set a Redis maxclients limit with appropriate alerts to catch connection storms before they breach thresholds.
- Enable verbose Daphne/Channels access logging to capture WSREJECT reasons for future debugging.
- Add a CloudWatch alarm on NewConnections rate-of-change to detect reconnect churn earlier than CurrConnections threshold.
The reject loop was amplified because the client retried immediately on every WSREJECT with no backoff, turning a single rejected route into roughly 21 new Redis connections a second. Adding backoff and a retry cap to the websocket client, alongside a Redis maxclients limit with its own alert, bounds how far a reject can propagate before it looks like this incident again. The other gap is diagnostic: verbose Daphne/Channels logging that records why an upgrade was rejected, not just that it was, would have let this investigation name the trigger instead of leaving it as an open question.
Frequently asked questions
Why did ElastiCache CurrConnections spike if Redis CPU and memory were healthy?
Redis metrics stayed well within safe limits throughout: CPU averaged 6.57 percent (max 9.26 percent), EngineCPU 5.03 percent, and database memory 2.34 percent. If maxclients had been hit, CurrConnections would have plateaued at the limit instead of peaking at 1,988 and falling back down, so ElastiCache was a passive victim of client-side churn, not its cause.
How do you tell a connection leak from a reconnect storm in ElastiCache metrics?
A leak shows CurrConnections rising slowly with a low NewConnections rate. Here, NewConnections summed to 6,377 in the same 5-minute window that CurrConnections jumped 4.5x, meaning connections were being created and torn down rapidly rather than accumulating. That simultaneous spike in both metrics is the signature of a reconnect loop, not a leak.
What triggered the WebSocket reconnect churn on eaze-websocket-001?
eaze-channels application logs showed dense WSCONNECTING, WSCONNECT, and WSDISCONNECT cycles alongside repeated WSREJECT entries on /ws/expert_availability, at an implied rate of about 21 reconnects per second. Each reject caused an immediate client retry with no backoff, and each retry opened a new Redis connection, which is what drove the CurrConnections spike.
Was the exact reason for the WSREJECT rejections identified?
No. The exact reason Daphne or Channels rejected the WebSocket upgrade was not present in the available logs, so it is recorded as an open question rather than guessed at. Candidate triggers include an auth or token validation change, a CORS or allowed-hosts change, or a routing change, none of which could be confirmed from the data on hand.