AI SRE for Kubernetes

Sherlocks AI runs 16+ specialized agents on top of your Kubernetes cluster to investigate incidents end to end. When a pod crashes, a deployment stalls, or a node runs out of resources, the agents correlate signals across logs, metrics, events, and recent deploys to return a draft root cause analysis in minutes, delivered directly in Slack, ready for the on-call engineer to confirm or edit.

Install via Helm. Read-only access, with Watson running as pods inside your cluster.

Kubernetes is good at telling you that something is wrong and bad at telling you why. kubectl describe pod gives you an exit code and a restart count. The cause is usually somewhere else: in the diff of the deploy that went out twenty minutes earlier, in a node that is short on disk, in a query that read more rows than anyone expected, or in a dependency the pod cannot reach.

So the on-call engineer becomes the correlation engine. Events from the scheduler and the kubelet, container logs, Prometheus or Datadog metrics, rollout history and CI all live in different tools, and each obvious first guess has to be checked and ruled out by hand before the real one is found.

That checking is the part Sherlocks AI takes over. The agents query each signal source in parallel, test the competing hypotheses against the evidence, and return the root cause along with the hypotheses they ruled out, so the engineer can verify the reasoning instead of redoing it.

The agents are trained on real production patterns. These are the failure modes handled out of the box, each group shown next to real investigations from the Kubernetes examples, and the RCA guide and upstream documentation for each.

1. 1Step 1: Deploy the Watson data agent inside your cluster.

   Install via Helm. Watson runs as a pod in a dedicated namespace, with a read-only ClusterRole scoped to what the agent needs: pods, deployments, events, logs, metrics.

2. 2Step 2: Connect your observability stack.

   Watson reads from your existing observability tools using service accounts. Native integrations for Datadog, New Relic, Prometheus, OpenTelemetry, and in-cluster Kubernetes events.

3. 3Step 3: Install the Slack app.

   Agents post investigation summaries directly in your incident channel. Engineers ask follow-up questions in natural language.

4. 4Step 4: Fire an alert.

   When Prometheus, Datadog, or any alerting tool fires, Watson spins up a parallel investigation across 16+ specialized agents, each querying a different signal source. A draft RCA lands in Slack within minutes.

Running somewhere else? The cloud pages cover AWS, Google Cloud and Azure, and we will walk your platform team through any other cluster on a call.

Watson reads from inside the cluster and from your connected observability tools.

This is a published case from the examples section, not a mock-up. The pod was OOMKilled, and the obvious first guesses were wrong. Here is how the investigation got from the alert to the line of code.

1. 1The alert
   CriticalKubernetes / prod-india-exchange2026-04-17 10:39:48 UTC

   Pod support-dashboard-staging-7cc9fd7874-gpf82 OOMKilled

   137 (OOMKilled)

   Exit Code

   2,500 MB

   Memory Limit

   22.36M rows

   KycStatusLog
2. 25 hypotheses ruled out
   • ✕

     Node-level memory pressure / noisy-neighbor eviction

     Node was at 3.12% of 30,808 MB allocated and the kill carried Reason=OOMKilled / Exit 137, which is a cgroup limit breach, not an eviction.

   • ✕

     Deployment / config regression introduced the OOM

     No rollout at incident time; start command and memory settings unchanged (start_gunicorn.sh, 1500M/2500M). Favors a runtime workload trigger.

   • ✕

     Gradual memory leak (recurring) rather than a one-shot spike

     RESTARTS=1 (a single event), sibling pods at 0 restarts, and the pod recovered and stayed Running. Event-driven, not a leak.

   • ✕

     Memory leak in the show_manual_review_kyc endpoint

     Memory is garbage collected properly when small result sets are returned; the issue is the size of the single result set.

   • ✕

     Background task consumed excessive memory

     No background tasks were running on the pod at the time of the crash.

3. 3Root cause

   Unbounded query over 22.36M KycStatusLog rows breached 2.5 GB

   KycStatusLog holds 22,364,964 rows. At ~100 bytes/row that is ~2.1 GB in-memory; with ~400 MB runtime overhead it consumes the entire 2500M budget. The unbounded KYC query materialized the full rowset and tripped the OOM killer. Root cause; repeatable until the path is paginated/streamed.

4. 4The fix
   • Paginate the show_manual_review_kyc path with an explicit page size (for example 500 rows) so no single request can materialize the full table.
   • Use QuerySet.iterator() with a bounded chunk_size for any code path that must stream large result sets, avoiding full-rowset materialization in the worker.

For the general method, read how to fix an OOMKilled container (exit code 137), or browse every Kubernetes investigation.