How to debug Kafka consumer lag in Kubernetes?

Kafka consumer lag is simple to define and easy to misdiagnose. Lag is how far behind your consumers are: the difference between the newest message on a partition and the offset your consumer group has committed. Rising lag means messages are arriving faster than you are processing them, or not being processed at all. The trap is that the obvious fix, add more consumers, is often useless and sometimes makes it worse. This guide covers why lag climbs when everything looks fine, and what to do instead.

What is Kafka consumer lag?#

Every partition in a Kafka topic has a log end offset, the position of the newest message. Your consumer group commits an offset as it processes. Lag is the distance between the two, measured per partition and summed across the group.

A little lag is normal and healthy, it just means messages are in flight. The problem is lag that climbs and does not come back down, because that means your consumers are permanently falling behind. Lag is tricky in Kubernetes specifically because consumers run as pods, and pods restart, scale, and get rescheduled constantly. That interacts with the consumer group protocol in ways that produce lag without any single consumer being slow.

Why is lag climbing when every consumer looks healthy?#

This is the situation that confuses people. Every consumer pod is Running, CPU looks fine, no errors in the logs, and yet lag keeps climbing. The reason is that lag is a property of the group, not of any one consumer, and three of its most common causes never show up as an unhealthy pod.

The group total hides all of them, so start by breaking it apart. This is the one command to run first:

kafka-consumer-groups.sh --bootstrap-server <broker> \  --describe --group <consumer-group> 

Read the output as three questions at once. Is the LAG column concentrated on a few partitions or spread evenly? Does every partition have a CONSUMER-ID, or are some unassigned? Is the same consumer holding the same partitions from one run to the next? Skew, a stalled rebalance, and a churning group each answer one of those differently, and the total tells you none of it.

If a partition shows no owner at all, the group is mid-rebalance rather than slow, and no amount of consumer capacity will help until it settles.

What is partition skew, and how does it cause lag?#

Kafka distributes work by assigning partitions to consumers, and a consumer can only read the partitions assigned to it. If your messages are unevenly distributed across partitions, one partition ends up with far more traffic than the others. The consumer handling that hot partition falls behind while the others sit nearly idle.

This is partition skew, and it produces a very specific signature: total lag is high but concentrated on one or two partitions while the rest are near zero. Sort the output to see it immediately:

# Partitions ranked by lag, worst firstkafka-consumer-groups.sh --bootstrap-server <broker> \  --describe --group <consumer-group> \  | awk 'NR>1 && $6 ~ /^[0-9]+$/ {print $6"\t"$2"-"$3}' | sort -rn | head 

If one partition carries almost all the lag, adding consumers will not help, because no new consumer can be assigned a partition that is already taken. The fix is upstream, in how messages are keyed: skew almost always traces to a partitioning key with low cardinality or one dominant value, such as keying by tenant when one tenant produces most of the traffic.

What is a rebalance storm, and why does pod churn trigger it?#

When a consumer joins or leaves a group, Kafka rebalances: it pauses everyone, reassigns partitions, and resumes. A single rebalance is brief. The problem in Kubernetes is that pods join and leave constantly, and each event triggers another rebalance.

During a rollout, old pods leave and new pods join, each one a rebalance. Under HPA scaling, pods are added and removed as load shifts, each one a rebalance. If pods churn faster than a rebalance completes, the group spends more time paused and reassigning than consuming, and lag climbs even though every consumer is technically healthy. It is not processing messages because it is stuck rebalancing.

The tell is in the consumer logs, and it is worth counting rather than eyeballing:

# How many times has this group rebalanced since the pods started?kubectl logs -l app=<consumer> -n <namespace> --tail=-1 \  | grep -cE '(Revoke|Attempt to heartbeat|Rejoining|Successfully joined group)' 

Then confirm what is churning the pods, because a rebalance storm is a symptom of pod instability rather than a Kafka problem:

# Pod ages: a tight cluster of young pods means a rollout just rankubectl get pods -l app=<consumer> -n <namespace> \  --sort-by=.metadata.creationTimestamp -o wide 

A set of pods all aged a few minutes, during a lag spike, is a rollout mid-flight. Widely varying ages with restarts is crash churn. Both produce the same rebalance storm from different directions.

Why does max.poll.interval.ms eject a live consumer?#

This one catches people because the consumer is alive the whole time. Kafka expects a consumer to call poll again within max.poll.interval.ms. If processing a batch takes longer than that interval, Kafka assumes the consumer is dead, even though it is actively working, and ejects it from the group. That triggers a rebalance, its partitions get reassigned, and when it finishes its slow batch and tries to commit, it has already been kicked out.

The result is a loop: the consumer processes slowly, gets ejected mid-batch, rejoins, triggers a rebalance, and the messages it was working on get reprocessed by someone else. The log line names it directly:

kubectl logs -l app=<consumer> -n <namespace> \  | grep -i 'max.poll.interval.ms\|leaving the group' 

The fix is a ratio, not a constant. What matters is max.poll.records multiplied by per-message processing time staying comfortably under max.poll.interval.ms. Lowering the record count is usually safer than raising the interval, because a longer interval also delays detection of a genuinely dead consumer.

Why does adding replicas sometimes make lag worse?#

The instinct when lag climbs is to scale up consumers. In Kubernetes that is often useless and sometimes harmful, for two reasons.

First, a consumer group can never have more active consumers than partitions. If your topic has 12 partitions and you scale to 20 consumers, 8 sit idle with no partitions to read. Scaling past the partition count does nothing for throughput. Check the ceiling before you scale into it:

# Partition count: the hard cap on useful consumerskafka-topics.sh --bootstrap-server <broker> --describe --topic <topic> \  | grep -c 'Partition:' 

Second, and worse, every consumer you add or remove triggers a rebalance. If lag is being caused by a rebalance storm in the first place, scaling up throws more rebalances onto the fire. The group pauses again, reassigns again, and falls further behind. You scaled up to fix lag and made it worse by adding instability. This is why the first question is always what is causing the lag, not how many consumers to add.

How do you debug Kafka consumer lag step by step?#

Work it in order, because each step rules out a cause.

Check per-partition lag first. Concentrated on one partition means skew, and scaling will not help. Spread evenly, keep going.

Check the rebalance rate. Frequent join and leave messages mean a storm, and the question becomes what is churning the pods: a rollout, HPA scaling, crashes, or evictions.

Check for poll-interval ejections. Logs about leaving the group for exceeding max.poll.interval.ms point at slow batch processing, not consumer count.

Check whether the pods are actually running at full speed. A CPU-throttled consumer processes slowly, which both falls behind and can trip the poll interval. Throttling is invisible in kubectl top, which shows usage, not the stalls:

# Share of scheduling periods the cgroup was throttled. Above ~0.2 is real.sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{pod=~"<consumer>.*"}[5m]))/sum by (pod) (rate(container_cpu_cfs_periods_total{pod=~"<consumer>.*"}[5m])) 

A pod sitting just under its CPU limit with a high throttle ratio is being stopped for part of every scheduling period, so its real throughput is well below what its usage suggests. The same resource-ceiling reasoning applies to memory, covered in our OOMKilled guide.

Check whether autoscaling has any room left. An HPA already at maxReplicas is over target and unable to act, which is easy to miss because it reports no error:

kubectl get hpa <consumer> -n <namespace> 

REPLICAS 25/25 against a target it is exceeding means the one automatic recovery lever is already pinned.

Only after those, consider scaling, only up to the partition count, and only once the group is stable enough that adding a consumer will not trigger another storm.

How do you prevent Kafka consumer lag?#

Size partitions for peak throughput and for the maximum useful consumer count, since consumers can never exceed partitions. Keep max.poll.records times per-message processing time well inside max.poll.interval.ms, so a slow batch never trips the timeout.

Then stabilise the pod set, because most of the causes above are pod churn wearing a Kafka costume. Give consumers enough CPU that they are not throttled, and bound how many can restart at once with a PodDisruptionBudget, so a rollout causes one small rebalance rather than a storm:

apiVersion: policy/v1kind: PodDisruptionBudgetmetadata:  name: notification-consumerspec:  maxUnavailable: 1  selector:    matchLabels:      app: notification-consumer 

Finally, scale on the signal that actually matters. CPU-based autoscaling can sit below target while a group falls steadily behind, because a latency regression raises cost per message without raising utilisation. KEDA scales on lag itself:

triggers:  - type: kafka    metadata:      consumerGroup: consumer-group-notifications      topic: production_notifications      lagThreshold: "1000" 

Monitor per-partition lag and rebalance rate together, ideally through open standards like OpenTelemetry, so you see skew or a storm forming before lag runs away. On managed platforms the same pod-churn dynamics apply, per the Amazon EKS documentation.

What does a real Kafka consumer lag incident look like?#

Sherlocks AI investigated one on a notification consumer, written up as the Kafka consumer lag RCA. Lag on consumer-group-notifications reached 35,942 messages against a 30,000 alert threshold. No single cause explained it.

The brokers were cleared first: all healthy, ISR stable at 3 for every partition, zero under-replicated. Ingest was flat. So the constraint was consumer capacity, and four ceilings turned out to be stacked at once. A rollout about 15 minutes before the alert left 25 pods aged 2 to 14 minutes, and the rebalancing suspended consumption on the affected partitions. The pods were running at 897 to 912m against a 1000m CPU limit with a CFS throttle ratio near 0.8, meaning they were stopped for roughly 80 percent of scheduling periods and had no burst headroom. The new revision had also raised p99 processing latency from 12ms to 45ms, a real regression but not enough on its own. And the HPA was at 25 of 25 replicas while reporting 89 percent against an 80 percent target: it wanted to scale and could not.

That last one is what made it an incident rather than a blip. Each of the first three degrades drain rate, and any of them alone would likely have been absorbed. The HPA cap removed the only automatic recovery lever, so instead of catching up, the group stayed behind. The durable fix was not more CPU or more replicas but scaling on lag rather than CPU, plus a PodDisruptionBudget so the next rollout causes a smaller rebalance.

That is the pattern with consumer lag. The nastiest cases are rarely one problem: they are a rebalance trigger, a processing slowdown, and a scaling constraint reinforcing each other, which is exactly why the group total tells you so little. More worked investigations are on the Kubernetes examples page.

Kubernetes Kafka consumer lag FAQ#

Why is lag climbing when my consumers are healthy? Usually partition skew, a rebalance storm from pod churn, or a consumer ejected for exceeding max.poll.interval.ms.

Does adding more consumers reduce lag? Only up to the partition count, and only if the group is stable. Beyond that, extra consumers sit idle, and adding them during instability triggers more rebalances.

What is a rebalance storm? Pods joining and leaving faster than rebalances complete, so the group spends more time reassigning partitions than consuming.

Why does max.poll.interval.ms eject a working consumer? If processing a batch takes longer than the interval, Kafka assumes the consumer is dead and removes it, even though it is alive.

How do I know if it is partition skew? Check per-partition lag. If it is concentrated on one or two partitions, it is skew, and scaling will not help.

Can HPA scaling cause lag? Yes. Each scale event triggers a rebalance, so aggressive CPU-based scaling on a consumer group can add the instability that causes lag.

Why does CPU look fine while consumers are slow? Usage and throttling are different things. A pod just under its limit can still be throttled for most of every scheduling period.