AI SRE for Prometheus and Grafana

Sherlocks AI uses Prometheus, Grafana and Loki as signal sources for incident investigation. When an alert lands in Slack, 16+ AI agents correlate metrics, logs and traces across time and services, test hypotheses against the data, and deliver a draft root cause analysis in your incident channel.

The exporters you already run, such as node_exporter, kube-state-metrics, cAdvisor, blackbox and your own application metrics, are all evidence Watson can reason over. Nothing to re-instrument, and no duplicate metrics pipeline.

Prometheus is very good at telling you that a number crossed a line. The alert carries the metric, the labels and the threshold. It does not carry the reason, and the reason is usually in a different series: the deploy annotation on another dashboard, the node the pod landed on, the dependency whose latency moved two minutes earlier.

So the on-call engineer writes PromQL under pressure. They open the dashboard, widen the time range, guess at the next query, and rule things out one panel at a time. Every obvious first guess has to be checked by hand before the real one is found.

That checking is the part Sherlocks AI takes over. It generates hypotheses about the root cause, tests them against your metrics, logs and traces, and ranks them by likelihood and impact, so the engineer verifies the reasoning instead of redoing it.

The incident types Prometheus and Grafana alert on most. Each group sits next to published investigations that were opened from a Prometheus or Grafana alert, with the RCA guide and upstream documentation for each.

1. 1Step 1: Connect Prometheus, Grafana and Loki.

   Add them as integrations alongside the rest of your stack. All permissions are read-only.

2. 2Step 2: Install the Slack app.

   Sherlocks AI monitors your Slack workspace for incident conversations, questions and alerts, so the Alertmanager and Grafana alerts your team already sends to Slack are where an investigation starts.

3. 3Step 3: An alert fires.

   An investigation is triggered manually, by the alert, or proactively. The Watson agent reads metrics from Prometheus and the other connected tools, then Sherlocks AI generates hypotheses, tests them against the data, and ranks causes by likelihood and impact.

4. 4Step 4: The RCA lands in Slack.

   The primary root cause with a confidence level, contributing factors, a timeline, recommended remediation, and links to the relevant logs, metrics, commits and dashboards. Engineers ask follow-up questions in the same thread.

Running Prometheus on Kubernetes? The Kubernetes page covers the in-cluster side: pod states, events and rollouts.

This is a published case from the examples section, not a mock-up. A Prometheus Node Exporter alert said host memory was above 95%, and the obvious first guesses were wrong. Here is how the investigation got from the alert to the code.

1. 1The alert
   CriticalPrometheus Node Exporter + Grafana2026-03-28 03:14 IST

   Host memory usage exceeded 95% on prod-vm-03

   14.2 GB

   JVM Heap

   8 min

   Downtime

   1,247

   Failed Txns
2. 24 hypotheses ruled out
   • ✕

     Traffic spike drove the heap up

     Heap growth is monotonic from 21:00 and is not correlated with any request pattern. prod-vm-01/02 carry the same API traffic and sit at 62% / 58%.

   • ✕

     Infrastructure fault or noisy neighbor on prod-vm-03

     dmesg shows a clean OOM kill of PID 2847 (java) with anon-rss 14.2 GB, accounting for nearly all of the 15 GiB host. No hardware fault.

   • ✕

     -Xmx8g heap is just sized too small for the workload

     The heap blew past -Xmx8g (8.9 GB by 01:00, incl. off-heap) and kept climbing to 14.2 GB. The growth is unbounded, so a bigger heap would only delay the OOM. Sizing is contributing, not the cause. Ruled out as root.

   • ✕

     An application thread-local cache is unbounded

     No significant thread-local retention was found; the retention is rooted in the Hibernate session.

3. 3Root cause

   Batch reconciliation job loads 4.2M rows in one session without clear()

   The nightly job runs findAll() over 4.2M PaymentTransaction rows in a single Hibernate session and never calls flush()/clear(), so the L1 cache pins every entity for the full 6h12m loop. The scope was changed 2 weeks ago from a 30-day window (~200K rows) to findAll(), a ~20x jump that turned a latent pattern into an OOM.

4. 4The fix
   • Add entityManager.flush() and entityManager.clear() every 500 records in the batch reconciliation loop.
   • Revert to the 30-day date filter or implement pagination (Pageable) to limit loaded entities per iteration.

For the general method, read how to debug a JVM memory leak and OOM kill, or browse every published investigation.