Alert info
- Severity
- Warning
- Detected by
- Prometheus
- Alert
- Consumer lag exceeded 30K threshold on production_notifications
- Time
- 2026-03-03 14:42 IST
- Service
- notification-consumer
consumer-group-notifications trails production_notifications by 35,942 messages. Investigate why the consumer side cannot keep up.
Rollout churn combined with CPU throttling and HPA hard-cap prevented consumer recovery, causing 35.9K message lag.
- Consumer Lag
- 35.9K msgs
- CPU
- 900m (limit)
- Replicas
- 25/25
Entities identified
Infrastructure
Kafka brokers
cluster
3-broker cluster. All healthy, ISR count stable at 3 for all partitions. No under-replicated partitions. Not implicated.
CPU limit
1000m cgroup
1000m hard limit for container CPU usage.
HPA
maxReplicas 25
Horizontal Pod Autoscaler configured with maxReplicas 25.
deployment rollout
~15m before alert
Deployment rollout managing pod revisions.
Service
notification-consumer
deployment
Owns message processing for the notification consumer group.
consumer pods
25 replicas
25 pods aged 2m-14m at alert time (rolling-restart pattern). Rebalancing suspended consumption on affected pods.
Data
production_notifications
Kafka topic
Source topic for production notifications.
consumer-group-notifications
consumer group
Consumer group on
production_notifications. TOTAL LAG 35,942 messages, well above the 30K alert threshold.
How they relate
The same relationships as text
- production_notifications hosted on Kafka brokers
- production_notifications consumed by consumer-group-notifications.
consumer-group-notificationstrailsproduction_notificationsby 35,942 messages across partitions. - consumer-group-notifications assigned to notification-consumer
- notification-consumer runs as consumer pods
- consumer pods throttled by CPU limit. CFS quota applies to the pods.
- HPA scales consumer pods. HPA scales the deployment based on CPU utilization.
- deployment rollout restarted consumer pods. Rollout triggers pod restarts.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Producer-side spike increased inbound message volumeRuled out
Pull producer rate on production_notifications for the past hour and compare against lag growth.
Producer rate steady at ~500 msg/sec for the past hour while lag climbed to 35.9K. Ingest is flat; this is a consumer-side problem.
Inbound message rate on production_notifications across the incident window.
metric value baseline
produce rate ~500 msg/s ~500 msg/s
window last 1h (flat)
Lag grew while ingest was flat. Not a producer spike. Ruled out.
Kafka broker failure or partition under-replicationRuled out
Check broker health and ISR counts for all partitions of production_notifications.
All brokers healthy, ISR count stable at 3 for every partition, no under-replicated partitions. Fetch availability is fine.
ISR counts and under-replicated partition count across the cluster.
metric value
brokers healthy 3/3
ISR per partition 3 (stable)
under-replicated 0
No broker crash or leader churn. Fetch path is healthy. Ruled out.
Code regression alone (latency increase) is the sole causeRuled out
Compare p99 per-message processing latency before and after the rollout.
p99 rose 12ms to 45ms post-rollout, a real regression, but a 3.75x latency bump on 25 running replicas at ~500 msg/sec cannot alone produce 35K lag without lost capacity. Contributing, not sole cause. Ruled out as standalone.
p99 processing latency per message before vs after the rollout.
metric before after
p99 12 ms 45 ms
Real regression and a contributing factor, but insufficient alone. The capacity loss is elsewhere.
Throughput collapsed under rollout churn + CPU throttle + HPA hard-capConfirmed
With ingest flat and brokers healthy, decompose consumer capacity: rollout state, CPU/throttle, and HPA headroom at alert time (14:42 IST).
All three throughput limiters are present simultaneously. Each degrades drain rate; together they hold lag above the 30K threshold. Confirmed; narrow into the individual limiters.
Total lag for consumer-group-notifications across partitions.
GROUP TOPIC LAG
consumer-group-notifications production_notifications 6324
consumer-group-notifications production_notifications 5564
consumer-group-notifications production_notifications 6131
...
TOTAL LAG: 35,942
Consumer cannot drain. Ingest is flat, so capacity is the constraint.
Rollout caused rolling restarts and consumer-group rebalancingConfirmed
Check rollout status and pod ages for notification-consumer.
A new revision rolled out ~15 min before the alert. 25 pods aged 2m-14m confirm a rolling restart; rebalancing suspended consumption on affected partitions, dropping drain rate during catch-up.
Deployment rollout state and pod creation timestamps.
deploy/notification-consumer successfully rolled out (~15m ago)
25 pods, ages 2m -> 14m (rolling-restart pattern)
Rebalancing temporarily unassigns partitions, reducing throughput. Confirmed contributor.
CPU throttling held pods below full processing speedConfirmed
Pull pod CPU usage and the CFS throttle ratio from Prometheus.
Pods run 897-912m against a 1000m limit with a throttle ratio ~0.8. The cgroup is throttled 80% of scheduling periods, so there is no burst headroom to catch up after rebalancing.
Per-pod CPU usage against the 1000m limit.
pod cpu mem
notification-consumer-xxxxx 897m 256Mi
notification-consumer-yyyyy 912m 248Mi
notification-consumer-zzzzz 903m 261Mi
limit 1000m
Pods pinned at ~900m against the 1000m ceiling. No headroom.
container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total.
throttled_periods / periods ~= 0.8
Throttled 80% of periods; processing stalls four-fifths of the time. Confirmed contributor.
HPA hard-cap at 25/25 blocked auto-scaling to absorb the deficitConfirmed
Read HPA status: current vs target CPU and replica count against maxReplicas.
HPA reports 89% CPU vs an 80% target with REPLICAS 25/25. It wants more replicas but maxReplicas=25 is saturated, so it cannot add capacity to drain the lag. This is the limiter that prevents recovery.
Current vs target CPU and replicas vs maxReplicas.
NAME TARGETS MINPODS MAXPODS REPLICAS
notification-consumer 89%/80% 5 25 25
Over target but capped at max. Auto-scaling cannot recover the deficit. Confirmed.
HPA target CPU threshold is too low, causing premature scaling limitsRuled out
Check if the 80% target is too low and forces unnecessary scale-outs that hit the cap early.
The 80% target is standard and appropriate for burst capacity. The issue is the hard cap preventing the needed scale-out, not the threshold being too low.
Multi-factor throughput collapse with HPA cap as the recovery blockerRoot cause
Combine the three limiters and the latency regression against the 35.9K lag.
Rollout churn cut drain rate, CPU throttle (~0.8) held pods below full speed, the post-rollout latency regression (12ms to 45ms) raised cost per message, and the HPA hard-cap (25/25) blocked the one recovery lever left. Together they pushed lag to 35,942, above the 30K threshold.
All limiters present simultaneously while ingest stayed flat.
factor state effect
rollout churn pods 2m-14m rebalance stall
CPU throttle ~0.8 ratio 80% stalled
HPA hard-cap 25/25 no scale-out
latency regression 12ms -> 45ms higher per-msg cost
result lag 35,942 > 30K threshold
No single factor suffices; their combination, with HPA blocking recovery, is the root cause.
Final RCA
Multi-factor throughput collapse with HPA cap as the recovery blocker
Rollout churn cut drain rate, CPU throttle (~0.8) held pods below full speed, the post-rollout latency regression (12ms to 45ms) raised cost per message, and the HPA hard-cap (25/25) blocked the one recovery lever left. Together they pushed lag to 35,942, above the 30K threshold.
Remediation
# Increase CPU limits to reduce throttling
kubectl set resources deploy/notification-consumer -n prod -c consumer --limits=cpu=2000m --requests=cpu=500m
# Raise HPA maximum to allow scaling
kubectl patch hpa notification-consumer -n prod -p '{"spec":{"maxReplicas":40}}'- Increase CPU limits from 1000m to 2000m to eliminate throttling and allow burst capacity.
- Raise HPA maxReplicas from 25 to 40 to provide scaling headroom.
- Investigate the p99 latency regression in the new code revision: consider rollback if lag does not recover.
- Implement consumer lag-based HPA scaling (KEDA) instead of CPU-based scaling for more responsive autoscaling.
- Configure PodDisruptionBudget to limit concurrent restarts during rollouts and reduce rebalancing impact.
No single limiter caused this lag; three independent ceilings, a rebalance from the rollout, a CPU throttle with no burst headroom, and an HPA already at its cap, happened to line up at once. Raising the CPU limit and the HPA maxReplicas removes two of those ceilings immediately, but the durable fix is scaling on the signal that actually matters: consumer lag itself, via KEDA, rather than CPU utilization, which can sit below target while a group falls behind on a slow-consumer regression. A PodDisruptionBudget on the consumer deployment bounds how many pods a future rollout can restart simultaneously, so the next deploy causes a smaller rebalance rather than a repeat of this one.
Frequently asked questions
Why did Kafka consumer lag reach 35.9K when the topic and brokers were healthy?
The consumer group's lag climbed to 35,942 messages while the producer rate held steady at about 500 messages per second and all three Kafka brokers stayed healthy with stable ISR counts. Three factors on the consumer side combined at once: a rollout 15 minutes earlier caused rolling restarts, pods ran at 897 to 912m against a 1000m CPU limit with an 80% throttle ratio, and the HPA was already capped at its maximum of 25 replicas.
How do rollout rebalancing, CPU throttling, and an HPA cap combine to cause lag?
No single factor was ruled sufficient alone. The rollout caused rebalancing that temporarily suspends consumption on affected partitions, CPU throttling at roughly 0.8 stalled processing 80% of scheduling periods, and the HPA reported 89% CPU against an 80% target but could not add replicas because it was already at its maxReplicas of 25. Together they held drain rate below the 30K lag threshold.
Was the post-rollout latency regression the cause of the lag?
A post-rollout code change raised p99 per-message processing latency from 12ms to 45ms, a real regression. On its own, a 3.75x latency increase across 25 fully running replicas at roughly 500 messages per second was judged insufficient to produce 35.9K lag, so it is treated as a contributing factor that compounded the capacity loss from rebalancing and CPU throttling, not the sole cause.
What fixes consumer lag caused by an HPA hard cap and CPU throttling?
Increase the CPU limit from 1000m to 2000m to remove the throttling, and raise the HPA maxReplicas from 25 to 40 so the autoscaler has headroom to absorb demand. Longer term, moving to consumer lag-based scaling with KEDA responds directly to lag instead of CPU, and a PodDisruptionBudget limits how many pods restart at once during a rollout, reducing the rebalancing hit.