Why did ElastiCache CurrConnections spike when nothing changed?

Connections to Redis sextupled in five minutes. Nobody deployed. Cache CPU is normal, memory is normal, hit rate is normal, and yet the alarm is real and the number on the graph is enormous. That combination is not a contradiction, it is the signature of a connection storm that started on the client side.

What does CurrConnections measure?#

The number of client connections currently open to the node, excluding replication connections. It is a gauge, sampled per minute, and it says nothing about whether those connections are doing work.

That last point matters. Ten thousand idle connections and ten thousand busy connections produce the same CurrConnections. The metric measures how many clients are attached, which is a property of your client fleet and its pooling behaviour, not a property of the cache's load. The full metric list is in the Amazon ElastiCache metrics documentation.

Why does the cache look healthy while connections spike?#

Because connection count and cache health are separate dimensions with separate ceilings.

Redis holds each connection in a client buffer, so connections consume file descriptors and a little memory, but they do not consume CPU unless they issue commands. A fleet that opens connections and sits idle produces a dramatic CurrConnections graph and a completely flat EngineCPUUtilization graph. Both are accurate.

The ceiling that actually matters is maxclients, which on ElastiCache defaults to 65,000. Below that, extra connections are an inefficiency. At it, new connections are refused and the failure becomes real and customer-visible. So the first question is never how high the number went, it is whether anything was turned away.

How do you tell a client storm from a cache problem?#

Ask the cache what it rejected. Two counters settle it.

REDIS=eaze-websocket-001.abc123.ng.0001.use1.cache.amazonaws.com redis-cli -h $REDIS INFO clients stats \  | grep -E 'connected_clients|blocked_clients|rejected_connections|evicted_keys|keyspace_' 

connected_clients:1788        # 280 -> 1,576.6 avg / 1,788 max in one 5-min bucketblocked_clients:0rejected_connections:0        # nothing was actually turned awayevicted_keys:0                # memory was never the constraintkeyspace_hits:98421093keyspace_misses:1120847 

rejected_connections:0 means the cache absorbed every one of them. evicted_keys:0 means memory was never pressured. At that point the cache has told you it is fine, and the investigation moves to whatever opened 1,500 sockets in five minutes.

Confirm the ceiling you were actually near, rather than assuming the default:

redis-cli -h $REDIS CONFIG GET maxclientsaws elasticache describe-cache-clusters --cache-cluster-id eaze-websocket-001 \  --query 'CacheClusters[].{Node:CacheNodeType,Engine:EngineVersion,Status:CacheClusterStatus}' \  --output table 

What causes a connection spike on the client side?#

Four patterns account for nearly all of them.

An HPA cycling replicas. Every new pod opens a fresh pool. A deployment going from 4 to 17 replicas with a 10 connection pool per pod adds 130 connections in the time it takes pods to become ready, and the old pods' connections linger until their timeouts expire, so the two overlap.

Pod churn from evictions or restarts. The same arithmetic, without the deliberate scale-up. A crash loop reconnects on every restart.

A reconnect loop. A client that fails, retries immediately, and fails again generates connections far faster than any scaling event. This is the one that actually reaches maxclients.

A pool with no ceiling. Some clients open a connection per request when the pool is exhausted rather than waiting. Under load that turns a traffic spike into a connection spike.

Correlate against the workload over the same window:

# Did replicas move? This is the usual answer.kubectl get events -n prod --sort-by=.lastTimestamp \  --field-selector reason=ScalingReplicaSet | tail -20 kubectl describe hpa eaze-channels -n prod | grep -A8 Events # Did pods restart rather than scale? Different cause, same graph.kubectl get pods -n prod -l app=eaze-channels \  -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,AGE:.metadata.creationTimestamp' 

HPA eaze-channels  4 -> 17 replicas at 07:54Z 

A scaling event at the same timestamp as the connection spike is the answer. The cache reported your deploy back to you.

How do you fix an ElastiCache connection spike?#

The fix belongs where the connections came from.

Cap and reuse the pool, so a replica count change moves connections linearly and predictably rather than in bursts. Concretely, that means a bounded maximum, an idle timeout shorter than the scale-down interval, and a wait rather than a new socket when the pool is exhausted.

# Bound the pool per replica so peak connections are a number you can compute:# replicas x poolSize, not "however many sockets the client felt like opening".env:  - name: REDIS_POOL_MAX    value: "10"  - name: REDIS_POOL_MIN_IDLE    value: "2"  - name: REDIS_POOL_IDLE_TIMEOUT    value: "30s"  - name: REDIS_POOL_ON_EXHAUSTED    value: "block" 

Add backoff on reconnect. An immediate retry loop is the one pattern that genuinely exhausts maxclients, and exponential backoff with jitter turns it into a survivable blip.

Smooth the scaling that triggers it. An HPA that jumps 4 to 17 is usually reacting to a metric with no stabilization window:

behavior:  scaleUp:    stabilizationWindowSeconds: 60    policies:      - type: Pods        value: 4        periodSeconds: 60 

Then alarm on the thing that means damage, not the thing that looks dramatic:

# CurrConnections as a fraction of maxclients, plus the counter that# actually indicates customer impact.aws cloudwatch put-metric-alarm \  --alarm-name eaze-websocket-001-connections-rejected \  --namespace AWS/ElastiCache --metric-name NewConnections \  --dimensions Name=CacheClusterId,Value=eaze-websocket-001 \  --statistic Sum --period 60 --evaluation-periods 2 \  --threshold 5000 --comparison-operator GreaterThanThreshold 

Resizing the node is the fix only when you are genuinely near maxclients and the connections are legitimate. Otherwise it buys headroom for a client-side bug and leaves it in place.

What does a real connection spike look like?#

Sherlocks AI investigated one on a WebSocket service. CurrConnections on eaze-websocket-001 went from about 280 to 1,576.6 average and 1,788 maximum inside a single five-minute bucket at 07:54 UTC, breaching a 1,500 threshold.

Cache health never moved. No rejected connections, no evictions, normal CPU and memory. The cause was HPA-driven pod churn on the eaze-channels deployment: replicas scaled and each new pod opened its own pool while the departing pods' connections had not yet timed out. Transient connection fan-out from scaling, not a deploy and not resource saturation. The full writeup is in the ElastiCache HPA connection burst investigation, and a related churn pattern on the same service is in the WebSocket churn investigation.

The lesson generalises: the cache is where a client-side event first becomes visible, which makes it the most commonly blamed and least commonly guilty component in this class of incident. When the workload doing the churning is itself under pressure, the Kubernetes pod eviction guide covers why replicas cycle in the first place, and Aurora load surges show the same downstream-symptom shape on the database side. The client-side view is documented in the Redis clients reference.

ElastiCache connection spike FAQ#

Is a CurrConnections spike always a problem? No. If rejected_connections is zero and the node is far below maxclients, the cache absorbed it and nothing was lost.

What is the actual connection limit on ElastiCache? maxclients, which defaults to 65,000. Check it directly rather than assuming, since it can be changed by a parameter group.

Why did connections spike without a deploy? An HPA scaling event, pod restarts, or a client reconnect loop all produce this without any code shipping.

Should I resize the cache node? Only if the connections are legitimate and you are genuinely near maxclients. Otherwise resizing hides a client-side pooling bug.

How do I stop it happening again? Bound the connection pool per replica, add backoff with jitter on reconnect, and give the HPA a stabilization window.

Why is cache CPU normal during the spike? Connections consume file descriptors and buffer memory, not CPU. Idle clients raise the count without doing any work.