Most AI SRE tools look great in a demo and struggle in a real incident. Before you buy one, ask the vendor eight things: does it connect to the tools and systems you already run, what does its MCP setup actually cover (the NSA has guidance on why that matters), does it remember your systems between incidents, what happens when its LLM provider goes down (Anthropic was out for 42 minutes in August), does it respect rate limits on the tools it calls, can it run inside your own network, is there a real permissions system rather than a prompt that says be careful, and what gets stripped out before your data leaves. Every question below links to the source behind it. Four tools answer all eight well: Sherlocks AI, Resolve AI, incident.io and Traversal, in that order.
Every AI SRE demo looks great. Most of them fall over the first time you use them for a real incident at 2am.
So we wrote down the eight questions that actually separate the useful tools from the demo-only ones. Send them to any vendor before you sign. If the answers get vague, that tells you what you need to know.
Under each question we say how Sherlocks AI handles it. Every claim links to its source, so you can check it yourself. At the end we rank the four tools that answer all eight well.
Want the shortlist first? Jump to the ranking. Want the whole market? See our top AI SRE tools in 2026 comparison.
1. Does it work with the tools you already have?
Nobody runs just one observability tool. It's Datadog and Prometheus. Or Splunk and Grafana. Plus PagerDuty. Plus Slack. An AI SRE that only connects to one of these won't help much.
And observability tools are only half the picture. Your database, your Kubernetes cluster and your AWS console are the other half. If the AI can read Datadog but can't talk to your Postgres directly, it will keep asking you for things it should just go and fetch itself.
Ask for the full integration list. Check both sides: the observability tools you use, and the systems those tools watch. If it only covers one side, it only does half the job.
How Sherlocks AI handles it
Sherlocks AI plugs into Datadog, New Relic, Prometheus, Kubernetes, and cloud APIs. The Watson data agent runs inside your VPC and talks to the underlying systems directly, so it doesn't have to wait for your observability layer to explain things to it.
2. Does its MCP setup actually cover what you need?
Almost every AI SRE vendor uses MCP (Model Context Protocol) to connect to outside tools. It's a real standard from Anthropic, and it works. But it has real gaps.
Two things to know:
MCP servers don't expose everything. Vendors ship the safe read operations first. Writes, admin actions, and deeper queries are often missing. The CData team wrote a good overview of what MCP is still missing today.
MCP has real security problems. The NSA put out formal guidance on MCP in May 2026 basically saying adoption ran ahead of security. Kong wrote up how MCP's default open-access model creates over-permissioned agents, and also how it messes with LLM performance once you expose too many tools at once.
Ask the vendor: which parts of my observability stack does your MCP setup actually cover? If they brush the question off, that's your answer.
How Sherlocks AI handles it
Sherlocks AI uses MCP where it's the right tool for the job, and uses direct integrations everywhere it isn't. Watson picks the deepest integration for each system, not the one that happens to be MCP-shaped.
3. Does it remember anything, or start from zero every time?
An AI that has to relearn your systems every time it starts is slow and expensive. It burns tokens rediscovering things it saw last week. It asks you the same questions over and over.
A knowledge graph fixes this. It keeps a map of your services, their dependencies and past incidents, so the agent starts with what it already knows.
A good one also learns from you. If you tell it in Slack that a service depends on a queue it missed, it should just remember. Forever. No form to fill in, no runbook to write. Ideally it should have spotted that dependency itself before you had to say anything.
Ask the vendor: what does your agent remember between investigations? How does it pick up corrections?
How Sherlocks AI handles it
Sherlocks AI builds a knowledge graph of your services, dependencies, deploys, and past incidents. Investigations start there. Corrections in Slack get pulled into the graph automatically, so the next investigation already has them.
4. What happens when Anthropic or OpenAI goes down?
If your AI SRE runs on one LLM provider, that provider's outages become your outages. This is not a hypothetical:
- On August 16, 2026, Anthropic went down for 42 minutes. Auth broke at 21:58 UTC. Everything was back by 22:40 UTC. Claude.ai, Claude Code, and Claude Cowork all down.
- StatusGator has logged 326 Anthropic outages since January 2025.
- The Anthropic status page has had multiple incidents in recent weeks including elevated errors and latency.
Making one tool work well across several LLMs is harder than it sounds. They all behave differently. Prompts that work on Claude break on GPT. Tool calls work differently too. Any vendor who says they support multiple LLMs should be able to show you they've tested it, not just say they built it.
Ask the vendor: which LLMs do you support today? And what happens on your side when Anthropic has a bad day?
How Sherlocks AI handles it
Sherlocks AI works with Anthropic Claude, OpenAI, AWS Bedrock, and open-weight models for teams that need to keep everything internal. If one provider goes down, Sherlocks AI falls back to another.
5. Does it play nice with rate limits on the tools it queries?
An AI SRE that hammers your observability stack with calls during an incident will get throttled. Here are the real numbers from the Datadog API rate limits docs:
- Event submission caps at 250,000 events per minute per org.
- The log ingestion API isn't rate limited, but plenty of others are.
- Hit a limit and you get a 429, then you wait until the reset window.
- Rate limit headers (X-RateLimit-Limit, X-RateLimit-Period, X-RateLimit-Remaining, X-RateLimit-Reset) tell you exactly where you stand.
In a demo, an AI that runs 100 Datadog queries to answer one question looks like magic. In production, it runs up your Datadog bill and gets rate-limited at exactly the moment you need it.
Ask the vendor: does your tool respect rate limits? Does it cache? Does it batch? Does it back off when it hits a 429?
How Sherlocks AI handles it
Sherlocks AI is built to be careful with the tools it queries. It respects rate limits and caches repeated queries within an investigation instead of re-fetching them.
6. Can you run it inside your own environment?
If you work in a regulated industry (finance, healthcare, government), your AI SRE can't be a SaaS product that sends your production data home. Even outside those industries, sending logs and traces to a third party means a security review nobody wants to sit through.
Deployment options that matter:
- Self-hosted. The whole product runs inside your cloud.
- BYOC (bring your own cloud). The vendor's software runs in your cloud account.
- Open-weight LLM support. So you're not forced to send data to Anthropic or OpenAI.
Not every vendor offers all three. Ask the exact question: can I run the whole thing inside my network with an open-weight model? If not, which part has to leave?
How Sherlocks AI handles it
Sherlocks AI runs self-hosted, or as a VPC deployment with the Watson agent read-only inside your cloud, or with open-weight LLMs for teams that can't send data to hosted providers. SOC 2 Type 2.
7. Is there a real permissions system, or does the vendor just tell the AI to behave?
This is where most AI SRE tools fall down.
In August 2026, a coding agent ran rm -rf on a developer's home directory while trying to test a sandbox. The sandbox didn't work. Everything was wiped. The agent's own safety checks were the very code being tested.
Telling the model to be safe is not a permissions system. A real one has three parts:
- Deny by default. The agent has zero permissions until you explicitly grant them. It doesn't inherit your admin access.
- Read-only for investigation. Anything the agent does to understand your systems needs zero write, execute, or delete permissions.
- Approval gates for writes. Any change to production needs a human approval the agent can't bypass, fake, or influence.
If the vendor's answer is that they tell the model to be careful, walk away.
How Sherlocks AI handles it
Sherlocks AI is read-only by default. Watson has zero write, execute, or delete permissions on your systems. Any write action needs explicit human approval, out-of-band, through a channel the agent can't touch.
8. What gets stripped before your data leaves your environment?
Your logs and traces are full of things you never meant to log. Personal data. Secrets a developer printed by accident. Internal customer IDs. If any of that reaches a hosted LLM prompt or the vendor's pipeline without being stripped out, you have a data exposure problem.
Ask the vendor: what patterns get stripped? Credit cards, JWT tokens, API keys, emails, phone numbers, SSNs? Is there a documented list, or is it best-effort?
The wrong answer is that the LLM won't do anything harmful with the data. The right answer is a specific list of what gets stripped, where it happens, and how you add your own patterns.
How Sherlocks AI handles it
Sherlocks AI runs a redaction layer before anything leaves your environment. Known sensitive patterns get stripped or replaced with placeholders. You can add your own custom patterns for internal identifiers.
The 8 questions in one place
Evaluating an AI SRE tool this quarter? Send the vendor these:
- Which observability tools AND which underlying systems do you integrate with directly?
- Which specific capabilities of my observability stack does your MCP integration actually cover?
- What does your agent remember between investigations, and how does it pick up corrections?
- Which LLMs do you support, and what happens on your side when Anthropic has a bad day?
- Does your tool respect rate limits, cache, batch, and back off on the tools it calls?
- Can I run the entire thing inside my network with an open-weight model?
- How does your permissions system prevent destructive actions, specifically, not by telling the model to behave?
- What patterns get stripped before data leaves my environment?
If any answer goes vague, you have your answer.
The 4 AI SRE tools that answer all 8 questions well
We looked at the tools on the market and scored each one against the eight questions above, using their public docs. Four of them answer all eight in a way that would hold up in a security review. Here's the ranking, with a short summary first.
#1Sherlocks AI
The only tool on this list that covers all eight in one deployment.
Watson runs read-only inside your VPC. Sixteen-plus specialized investigation agents work in parallel across metrics, logs, traces, and deploys. Multi-LLM across Anthropic Claude, OpenAI, AWS Bedrock, and open-weight models, so a bad afternoon at Anthropic doesn't take Sherlocks AI down. Plugs into Datadog, New Relic, Prometheus, Kubernetes, and cloud APIs directly. The knowledge graph picks up corrections from Slack automatically. PII gets stripped before anything leaves your environment. SOC 2 Type 2. Free tier of 30 investigations a month.
Where it wins: all eight. Read-only Watson plus PII stripping makes for an unusually easy security review. Multi-LLM support is tested, not just announced.
Where it doesn't fit: if you want a full observability platform. Sherlocks AI sits on top of your observability stack. It doesn't replace it.
#2Resolve AI
The strongest architecture story after Sherlocks AI. Resolve AI coined the word harness that the whole category now uses (context, tools, reasoning, guardrails, evals). Their plugin for Claude Code, Cursor and Codex is genuinely different from what other vendors ship, and pulling your own SKILL.md files in as investigation context is a smart idea.
Where it wins: integrations, knowledge graph, permissions (human in the loop for write actions).
Where it falls short: their public docs are Claude-first, so support for other LLMs is unclear. Deployment is SaaS-first. Pricing is still in preview.
#3incident.io
Their AI SRE page is one of the few in the category that spells out the security answers: a separate Nexus model instance per customer, sandboxed containers for code analysis, redaction before anything reaches a model provider, and writes that only happen through pull requests. Refreshing.
Where it wins: permissions (PR-only writes), PII handling, isolation (per-customer model).
Where it falls short: SaaS only, and their observability integrations are narrower than Sherlocks AI or Datadog Bits.
#4Traversal
Their Production World Model and Causal Search Engine are the best answer to the knowledge graph question in the category. Their workers can join incidents on their own and hand off to each other. Knowledge Bank 2.0 lets you correct things in context instead of editing runbooks.
Where it wins: knowledge graph, correlation depth, worker autonomy.
Where it falls short: SaaS only (no BYOC in their public docs), and multi-LLM support isn't spelled out the way Sherlocks AI and Resolve AI spell it out.
Frequently asked questions
Ask for the full integration list. Check both sides: the observability tools you use, and the systems those tools are watching. If it only does one side, it's only doing half the job.
Ask the vendor: which specific capabilities of my observability stack does your MCP actually cover? If they wave you off, that's your answer.
Ask the vendor: what does your agent remember between investigations? How does it pick up corrections?
Ask the vendor: which LLMs do you actually support today? What happens on your end when Anthropic has a bad day?
Ask the vendor: does your tool respect rate limits? Does it cache? Does it batch? Does it back off when it hits a 429?
Ask the specific question: can I run the entire thing inside my network with an open-weight model? If not, exactly which part has to leave?
A real one has three properties: deny by default, read-only for investigation, and approval gates for writes. If the vendor's answer is we tell the model to be careful, walk away.
Ask the vendor: what patterns get stripped? Credit cards, JWT tokens, API keys, emails, phone numbers, SSNs? Is there a documented list, or is it best-effort?
Related reading
Top 13 AI SRE Tools in 2026
The fuller comparison this checklist was written to sit beside: causal depth, autonomy, Kubernetes fit and pricing across the whole market.
What Is AI SRE?
The plain-language explanation of the category, how it differs from AIOps, and the four levels of autonomy.
Why AI Agents Fail in Production
The six layers where agents break, and why the failures rarely look like errors.
Resolve AI vs Sherlocks AI
The head-to-head with the number two on this list.
See an AI SRE work a real incident
Book 30 minutes with our team and watch an investigation run on your own stack.
Book a demo →