AI SRE for Kafka
Turn consumer lag and broker alerts into a draft root cause analysis in minutes. Watson reads your consumer groups, topics and brokers read-only, correlates the lag with the consumer pods, autoscalers and deploys behind it, and posts the root cause in Slack with the evidence attached.
sre@prod-bastion ~ $ sherlocks investigate --source kafkaALERT SIGNAL CORRELATEDconsumer lag 35,942 msgs HPA capped 25/25cpu throttle ~0.8 ratio 1000m limitp99 per message 12ms -> 45ms rollout 15m agoisr per partition 3/3 stable brokers ruled out sre@prod-bastion ~ $ sherlocks explain consumer-lag
Why Kafka alerts are hard to explain
A lag alert names a topic and a number. It does not say whether the producers sped up, the brokers faltered or the consumers slowed down, and the cause usually sits in one of three places.
Pods restarting through a rollout, CPU throttled against a tight limit, an autoscaler at its cap or a slower code path after a deploy. The topic looks fine and the group falls behind anyway.
Rebalances pause consumption, partition assignment caps how many consumers can work, and a skewed key puts most of the traffic on one partition. None of it shows as an unhealthy pod.
Under-replicated partitions, leader churn or a disk filling up. Quick to confirm or rule out from ISR counts, and ruling it out is what points the investigation at the consumers.
So the on-call engineer runs kafka-consumer-groups --describe, checks the ISR counts, opens the pod list and the autoscaler, pulls the deploy history, and rules out guesses one at a time. Sherlocks AI takes over that checking: 16+ agents test hypotheses against the data and rank them, so the engineer verifies the reasoning instead of redoing it.
Kafka incidents, worked end to end
Real published investigations, grouped by failure class, with the RCA guide and upstream documentation for each.
Consumer lag
- Lag climbing while every consumer is healthy
- Capacity lost to CPU throttling
- HPA at maxReplicas with lag still rising
- Latency regression after a deploy
- Producer burst outrunning the group
Read the producer rate before you blame the consumers, and read per-partition lag before the group total. One published case fell 35.9K messages behind with ingest flat at about 500 messages a second and all three brokers healthy: a rollout 15 minutes earlier caused a rebalance, pods sat at 900m against a 1000m CPU limit with an 80% throttle ratio, and the HPA was already at 25 of 25 replicas, so nothing could add capacity.
Other brokers: RabbitMQ, SQS and Service Bus
- Consumer cannot reach the broker at startup
- Queue depth growing
- Dead letter queue filling
- Message age climbing
The queue is often the victim rather than the cause. A Celery consumer crash-looped after its startup failed to open an AMQP connection to RabbitMQ, so the first alert was CrashLoopBackOff on the pod rather than anything on the broker. Sherlocks AI reads the RabbitMQ management API, SQS queue attributes and CloudWatch, and Service Bus queue and topic metrics, all read-only.
Other Kafka failure modes
What to look for in each, from the Kafka documentation and our consumer lag guide.
Rebalance storms
- Rolling restart churn
- HPA scaling during instability
- max.poll.interval.ms ejecting a live consumer
- Session timeouts under GC pauses
- Partitions left unassigned
Every consumer that joins or leaves a group triggers a rebalance that pauses consumption while partitions are reassigned. In Kubernetes pods join and leave constantly, so a rollout or an autoscaler can keep a group rebalancing faster than it can settle. A partition with no CONSUMER-ID in the describe output usually means the group is mid-rebalance, or that there are more partitions than consumers. Either way, it is not a slow-consumer problem, and adding consumers makes it worse.
Partition skew
- One hot partition holding most of the lag
- Low-cardinality partitioning key
- More consumers than partitions
- Idle consumers next to a saturated one
A consumer can only read the partitions assigned to it, and a group can never have more active consumers than partitions. If lag is concentrated on one or two partitions while the rest sit near zero, the cause is upstream in how messages are keyed, and no new consumer can take over a partition that is already owned.
Broker and replication health
- Under-replicated partitions
- Leader elections and controller failover
- Broker disk filling
- ISR shrinking under load
- Offline partitions
The broker side is where a lag investigation should start and usually finish quickly. The ISR count per partition, the under-replicated partition count and the broker count tell you whether the fetch path is healthy. In the published lag case they were, 3 of 3 brokers with ISR stable at 3, which is what moved the investigation to the consumer side.
Producer failures
- Request timeouts and retries
- acks=all waiting on a slow replica
- Record too large
- Buffer exhaustion under backpressure
Producer errors surface in the application's logs rather than in the broker's, and a retry storm can look like a traffic spike from the broker's side. The producer rate against the lag curve is the first check: if ingest is flat, the problem is not the producer.
A real investigation, step by step
A published case from the examples section, not a mock-up. A consumer group fell 35.9K messages behind on a production topic while the producers and all three brokers stayed healthy. Here is how the investigation got from the lag alert to three limiters that lined up at once.
- 1The alertWarningPrometheus2026-03-03 14:42 IST
Consumer lag exceeded 30K threshold on production_notifications
35.9K msgsConsumer Lag900m (limit)CPU25/25Replicas - 24 hypotheses ruled out
Producer-side spike increased inbound message volume
Producer rate steady at ~500 msg/sec for the past hour while lag climbed to 35.9K. Ingest is flat; this is a consumer-side problem.
Kafka broker failure or partition under-replication
All brokers healthy, ISR count stable at 3 for every partition, no under-replicated partitions. Fetch availability is fine.
Code regression alone (latency increase) is the sole cause
The latency regression narrowed the throughput margin from ~2,083 msg/sec to ~555 msg/sec against 500 msg/sec ingest, which is a real contributor. On its own though, the margin is still above ingest, so the regression cannot account for the full 35K gap. Combined with the CPU throttle and the HPA cap it does. Contributing, not sole cause.
HPA target CPU threshold is too low, causing premature scaling limits
An 80% CPU target is within the usual range for throughput-focused workloads. The issue here is the hard cap preventing the needed scale-out, not the threshold itself being wrong.
- 3Root cause
Multi-factor throughput collapse with HPA cap as the recovery blocker
Rollout churn cut drain rate, CPU throttle (~0.8) held pods below full speed, the post-rollout latency regression (12ms to 45ms) raised cost per message, and the HPA hard-cap (25/25) blocked the one recovery lever left. Together they pushed lag to 35,942, above the 30K threshold.
- 4The fix
- Increase CPU limits from 1000m to 2000m to eliminate throttling and allow burst capacity.
- Raise HPA maxReplicas from 25 to 40 to provide scaling headroom.
For the general method, read how to debug Kafka consumer lag in Kubernetes, or browse every Kubernetes investigation.
How it works
(04 Steps)- 01Connect Kafka, read-only
Access is consumer group and topic metadata. Consumer pods and autoscalers come through the Kubernetes integration.
- 02Install the Slack app
The lag and broker alerts your team already sends to Slack are where an investigation starts.
- 03An alert fires
Watson reads the group, the topic, the brokers, the consumer pods and recent deploys. Sherlocks AI tests hypotheses against the data and ranks the causes.
- 04The RCA lands in Slack
Root cause with a confidence level, timeline, remediation and links to the evidence. Ask follow-ups in the thread.
What Watson reads
Supported today: Apache Kafka, plus RabbitMQ, Amazon SQS and Azure Service Bus for teams running more than one broker. Consumers on Kubernetes are read through the Kubernetes integration, and SQS and Service Bus through the AWS and Azure integrations.
From Kafka. consumer group lag per partition, topic and partition metadata, ISR and under-replicated partition counts, broker health and produce rate. Read-only consumer group and topic metadata access.
From Prometheus. CFS throttle ratio, container CPU usage and topic produce rate. Read through the Prometheus and Grafana integration, and each appears as evidence in the published consumer lag investigation.
Around Kafka. kubectl rollout status, pod ages, kubectl top pods, HPA status and consumer APM latency. The consumers are usually where a lag alert starts, so the deploys, scaling events and per-message latency are read too.
Security and data handling
Read-only. For Kafka that is consumer group and topic metadata. Sherlocks AI cannot produce or consume messages, commit or reset offsets, create or delete topics, or change any configuration.
Message bodies stay where they are. The investigation works from lag, offsets, partition assignment and broker metrics. It does not read the contents of your messages.
Stays in your environment. Watson runs inside your network. Only aggregated metrics and metadata reach the Sherlocks AI platform, or nothing at all with the self-hosted deployment.
SOC 2 Type II. The Sherlocks AI cloud environment is audited every year against the SOC 2 Type II framework.
Why Sherlocks AI for Kafka incidents
In the published lag case the topic, the producers and the brokers were all healthy. Sherlocks AI reads the consumer pods, the autoscaler and the deploy history alongside Kafka, so the alert is traced to whatever is actually holding the group back.
Consumer group lag, pod ages, CPU throttle ratios, HPA status and per-message latency are read together, so a cause that spans three of them is found as one cause.
Consumer group and topic metadata, nothing that produces, consumes, commits offsets or reconfigures.
Every root cause comes with a confidence level and the evidence behind it, including the hypotheses that were ruled out.
Frequently asked questions
(07 Questions)Can Sherlocks AI investigate Kafka consumer lag?
Yes. The published investigations on this site include a consumer group that fell 35,942 messages behind on a production topic while producers and brokers were healthy. The investigation ruled out a producer spike, a broker fault and a code regression on its own, then traced the lag to rollout churn, CPU throttling and an HPA at its maximum replica count combining at once.
Which Kafka deployments does Sherlocks AI work with?
Apache Kafka, with consumer group lag, partition health, broker metrics and throughput read through read-only metadata access. RabbitMQ, Amazon SQS and Azure Service Bus are supported alongside it for teams running more than one broker.
Does Sherlocks AI need write access to Kafka?
No. All permissions are read-only. For Kafka that is consumer group and topic metadata access. Sherlocks AI does not produce or consume messages, and it cannot commit or reset offsets.
Why is consumer lag climbing when every consumer looks healthy?
Because lag is a property of the group, not of any one consumer. Partition skew, a rebalance storm from pod churn or autoscaling, and a live consumer ejected for exceeding max.poll.interval.ms all raise lag without a single pod reporting unhealthy. Read lag per partition before the group total, and check the rebalance rate before scaling.
Does adding consumers fix consumer lag?
Often not. A group can never have more active consumers than the topic has partitions, so extra pods beyond that sit idle. If the lag is concentrated on one hot partition the fix is in the partitioning key, and if the group is already rebalancing, adding pods during the instability makes it worse. In the published case the autoscaler had already hit its cap, and the recovery lever was CPU headroom and a higher maxReplicas.
How does a Kafka alert become an investigation?
Sherlocks AI monitors your Slack workspace for incident conversations and alerts. An investigation can be triggered manually, by an alert, or proactively. Watson reads the Kafka, Prometheus and Kubernetes signals, and Sherlocks AI generates hypotheses, tests them against the data, and ranks them by likelihood and impact.
How do I know whether to trust a root cause?
Every root cause comes with a confidence level, and every claim cites the logs, metrics or commits it rests on. The published investigations also show the hypotheses that were checked and ruled out.
Get started
Start free with 30 investigations per month, or book a 30-minute demo and we will walk you through a live incident replay on your own Kafka setup.
Questions? Get in touch.
Sherlocks AI is used by Fynd, Topmate, Wafeq, TradeIndia, and Lokal to investigate production incidents. SOC 2 Type II certified. Watson runs read-only.