GPU workloads fail differently from web services, and the failures are usually silent. A GPU does not crash and page you when it thermally throttles. It just runs 30 percent slower, at full price, with no alert anywhere. Traditional monitoring was built for request-response systems, not for hardware that quietly degrades across silicon, drivers, interconnects, schedulers, and frameworks. This blog walks through the full GPU failure surface, why each layer fails in ways standard observability misses, and how agentic SRE isolates a failure to the exact layer it lives in. If you are running or planning to run GPU workloads in production, this is the reliability gap nobody warned you about. For context on how investigation-first SRE works more broadly, see what AI SRE means in 2026.
$2–$7/hr
H100 rental cost
or $25K–$40K to buy outright
30%
Slower under throttling
with no error message anywhere
5–10×
Slower on NCCL fallback
at full price, no alert logged
18%
SM activity, starved GPU
allocated, running, mostly idle
Why Do GPU Workloads Need Different Reliability Thinking?
A single NVIDIA H100 GPU costs between $2 and $7 per hour to rent on major cloud providers, or $25,000 to $40,000 to purchase outright. (These figures are consistent across multiple independent pricing analyses as of mid-2026.) An 8-GPU training instance starts at $16 to $20 per hour just for the accelerators.
That economic fact changes what reliability means. On a typical web service, an incident is measured in downtime. On a GPU fleet, the cost of a card running at low utilization for a week can quietly outweigh the cost of a full outage. An idle or underutilized GPU does not care that it is idle. It bills the same whether it is training a frontier model or waiting for a slow data loader to feed it.
This is the core shift teams miss when they bring web-service reliability thinking to GPU workloads. The monitoring, the alerting, the investigation playbooks, everything was built for a world where failure means "the service is down." In GPU land, the most expensive failures are the ones where everything looks like it is running fine.
How Do GPU Workloads Actually Fail?
A web service fails in a few familiar ways. It runs out of memory, a dependency times out, a bad deploy ships. A GPU workload has a much taller failure surface, and the failures at each layer look different.
Walk up the stack.
The GPU failure stack — silicon to script
XID errors, ECC faults, thermal throttling, silent corruption
↓CUDA mismatches, missing Fabric Manager, bad MIG partitions
↓Idle-but-allocated GPUs, orphaned processes, queue starvation
↓NCCL falling back from RDMA/InfiniBand to slow TCP
↓OOMs, memory leaks, NaN loss, wasted GPU-hours
Most of these failures produce no alert. They just make things slower, more expensive, and quietly wrong.
The silicon
The GPU itself can throw XID errors, which are NVIDIA's device-level fault codes logged through the kernel driver. Some of these are fatal: XID 79 means the GPU has literally fallen off the PCIe bus and needs a node reboot. XID 48 means an uncorrectable double-bit ECC memory error, which kills the running job.
Others are quieter and more dangerous. A GPU can accumulate correctable ECC errors (XID 94) that do not crash anything immediately but indicate memory cells that are failing. Thermal throttling clocks the card down under sustained load, making everything 30 percent slower, with no error message anywhere.
The most insidious failure at this layer is silent data corruption from memory errors that are not caught by ECC. Research teams running large GPU clusters have documented cases where training runs continued without apparent issues while quietly producing corrupted results, only visible later when loss values stopped making sense. The application kept running. The numbers were garbage.
The driver and runtime
A container built against a CUDA version newer than the node's driver supports will fail on startup. A missing Fabric Manager makes an 8-GPU box unusable for NVSwitch-based communication. A misconfigured MIG (Multi-Instance GPU) partition hands the job the wrong memory slice.
These are not exotic edge cases. They are the most common "works on my machine" class of GPU failure, and they happen every time someone updates a container image or a driver without checking the compatibility matrix.
The scheduler
GPUs sit allocated but idle, held by a crashed process, a forgotten notebook, or a job that finished but did not release the device. New jobs queue behind them. The fleet looks fully utilized and is actually half working. Standard Kubernetes scheduling was not designed to handle the topology and allocation complexities of GPU hardware, which is exactly why NVIDIA donated its Dynamic Resource Allocation Driver for GPUs to the CNCF at KubeCon Europe in March 2026, signaling that GPU scheduling is now a first-class infrastructure problem.
The interconnect
This is often the single largest dollar-cost failure on a GPU fleet, and one of the hardest to catch.
In distributed training, NCCL (NVIDIA's collective communication library) can silently fall back from fast InfiniBand or RDMA to slow TCP if an environment variable points at the wrong network interface. The job still runs. It still produces output. It just runs 5 to 10 times slower, at full price, with no error in any log.
Nothing actually breaks. There is no alert to fire. There is just a much bigger bill and much slower progress, and a team that assumes the model simply got more compute-intensive than expected.
The framework and script
A batch size that fits comfortably on an 80GB card OOMs at a longer sequence length. A memory leak grows a gigabyte an hour and crashes at hour eight, after which the job restarts from scratch because the checkpoint interval was set to twelve hours. Loss goes NaN and the run burns GPU-hours producing nothing useful.
The through-line across every layer: most of these failures do not announce themselves. There is no red alert for "your distributed training job is using TCP instead of RDMA." There is just a bigger bill, slower progress, and a team that assumes everything is fine because no dashboard turned red.
A 2026 research paper specifically titled "When GPUs Fail Quietly" studied this exact pattern in production GPU telemetry and found that failures with differential observability, where the GPU suffers but the monitoring designed to notice it does not, are becoming the dominant failure mode as GPU workloads scale.
What Are the Two Real Problems to Solve?
Everything above collapses into two problems, and both are economic.
Downtime
A fault kills or stalls the job
- Card sits unproductive until diagnosed
- Goal is MTTR: back to useful work fast
- Certainty on hardware faults to pull the right node
Waste
The card is busy but far below capacity
- Nobody gets paged for low utilization
- Can persist quietly for weeks
- The bigger, quieter, more expensive problem
Both are economic problems. Neither shows up reliably on a standard uptime dashboard.
Downtime. A fault kills or stalls the job, and the expensive card sits unproductive until someone diagnoses it. The goal here is MTTR: get the GPU back to useful work fast, and when the fault is genuinely the hardware, know that with certainty so the right node gets pulled. We cover why investigation time dominates MTTR more broadly in why incident debugging is still slow in 2026.
Waste. The card is "busy" but running far below capacity, or allocated but idle. This is the bigger and quieter problem. Nobody gets paged for low GPU utilization, so it can persist for weeks. For a GPU cloud provider, realized utilization is the business. For an AI team, it is the difference between a training budget that lasts two months and one that lasts six.
Why Can't Traditional Monitoring Handle This?
You could, in principle, build a dashboard panel for every one of these failure modes. Teams that run GPUs at scale often do. The problem is that a panel shows a symptom. The diagnosis lives in the correlation between panels.
GPU utilization is low. Is that the data loader, the interconnect, a thermal throttle, or a straggler node? You cannot answer that from the utilization graph alone. You have to hold the whole stack in your head at once, form a hypothesis, and go check the specific signal that confirms or kills it.
This is the same investigation gap that exists in traditional SRE, the gap between knowing that something is wrong and understanding why. It is just more expensive here, because every minute of investigation on a GPU fleet is billed at the most expensive compute rate in the datacenter. The signals exist. They are just scattered across DCGM metrics, kernel logs, NCCL debug output, framework logs, and scheduler state, and nobody is correlating them.
What is needed is not a wider dashboard. It is an agentic workflow that knows where each layer's signals live and can reason across them. Start from the symptom, generate hypotheses, pull the evidence, isolate the failure to the exact layer it lives in. The output is not a chart. It is a root cause and a next action.
What Does "Isolate to the Exact Layer" Look Like in Practice?
Five concrete examples.
Hardware or batch size?
A training run crashes. The instinct is to blame the card. An agentic system checks XID and ECC records against the framework log and returns one of two clean verdicts. Either "XID 79, GPU fell off the bus on node gpu-07, genuine hardware fault, drain the node" or "no XID, no ECC, CUDA OOM at step 4,300, batch size exceeds memory at this sequence length, reduce batch or enable gradient checkpointing." One of those is a hardware ticket. The other never should have been.
The silent 5x-slower job.
A distributed run works but is inexplicably slow. The system notices NVLink and InfiniBand bandwidth counters are near zero while network TX is saturated, and that NCCL is using the socket transport instead of RDMA. Root cause: a config change broke the interface variable. Fix it, recover most of the throughput, save potentially thousands of dollars in wasted GPU-hours.
The starved card.
A GPU is allocated and running, but SM (streaming multiprocessor) activity averages 18 percent. Correlating GPU idle time against a pinned CPU and high disk-read latency points at a data loader that cannot feed the card fast enough. More workers, prefetch, or faster storage, and the same GPU does 3 to 4 times the work.
The straggler.
A 16-GPU distributed job is slow, but 15 of the GPUs are fine. Because the all-reduce synchronization runs at the speed of the slowest rank, every card looks degraded. The system identifies the one node lagging each step, traces it to a climbing ECC error rate, and recommends cordoning the node. Fix one card, recover the whole cluster.
The expensive late failure.
Instead of discovering an out-of-memory crash at hour eight, the system catches the memory trend climbing steadily from step zero, projects the OOM point, and flags that the checkpoint interval would lose the entire training run. An early, cheap fix replaces the worst failure mode in GPU training.
Why Does This Matter Even More for GPU Cloud Providers?
For a GPU cloud provider, isolation is worth even more, because it answers the question every support ticket really asks: is this your hardware or my job?
When the answer arrives automatically, with evidence, two things happen. Genuine hardware faults get escalated already diagnosed, so the right card gets pulled fast. And the large share of issues that were never the hardware get a clear, customer-side root cause instead of becoming a support ticket that ties up an engineer to disprove. The provider's realized utilization goes up, the support load goes down, and the customer gets a better answer than they would have gotten investigating on their own.
The Missing Layer
We spent a decade building the tooling to make CPU-bound, microservice reliability tractable. Tracing, structured logs, incident response platforms, SRE as a discipline. GPU workloads arrived faster than the equivalent tooling, and the economics make the gap more expensive than it ever was for web services.
The failure surface is taller. The failures are quieter. And every minute of downtime or waste is charged at the most expensive compute rate in the stack.
Agentic SRE is how that gap closes. Not another dashboard on top of the pile, but a layer that understands the whole stack from the silicon to the script, isolates a failure to the exact point it lives, and tells you what to do about it before the GPU-hours are gone. We cover the underlying telemetry this kind of investigation depends on in the four pillars of telemetry, and the broader pattern of why AI agents fail silently in why AI agents fail in production.
Key Takeaways
- •GPU reliability is a fundamentally different problem from web service reliability. The failure surface spans from silicon-level hardware faults to framework-level script errors, the failures are mostly silent, and the cost per wasted minute is dramatically higher.
- •The waste problem is often bigger than the downtime problem. Low utilization does not page anyone, so it persists for weeks. The most expensive GPU failures are the ones where everything looks like it is running.
- •Silent failures are the dominant mode. Thermal throttling, NCCL falling back to TCP, ECC errors corrupting training data: none of these throw alerts. They just make things slower, more expensive, and quietly wrong.
- •Dashboards show symptoms, investigation shows causes. When GPU utilization is low, the cause could be the data loader, the interconnect, a thermal throttle, or a straggler node. The answer lives in the correlation between signals, not in any single panel.
- •The tooling gap is real and being filled. We spent a decade building SRE tooling for CPU-bound workloads. GPU workloads outpaced the equivalent observability. Agentic SRE is the layer being built to close it.
Frequently Asked Questions
GPU failures often produce no errors at all. Thermal throttling, interconnect fallbacks, and silent memory corruption degrade performance invisibly, making them undetectable by monitoring designed to watch for crashes and error codes.
An XID error is NVIDIA's device-level fault code, reported through the GPU driver into the system kernel log. Different XID codes indicate different problems and require different responses, from a simple application restart to draining an entire node. NVIDIA maintains a full catalog at docs.nvidia.com/deploy/xid-errors.
In distributed training, NCCL can silently switch from fast InfiniBand or RDMA to slow TCP when a network interface is misconfigured. The job continues running, just 5 to 10 times slower, at full cost, with no error logged anywhere.
A dashboard shows individual metrics. Agentic SRE correlates signals across the full stack, from hardware fault codes through driver state, scheduler allocation, interconnect bandwidth, and framework logs, to isolate a failure to its exact layer and recommend a specific action.
No. Cloud GPU users face the same silent failures. The difference is that on the cloud, every minute of wasted utilization is billed directly, making the cost of missed diagnosis even higher.
Thermal throttling clocks a GPU down under sustained load to protect the hardware, silently making a job run up to 30 percent slower with no error logged, so it is easy to mistake for the model simply becoming more compute-intensive.
Silent data corruption happens when a GPU memory error is not caught by ECC, so a training run keeps producing output that looks normal until loss values or downstream results stop making sense, sometimes much later.
A GPU bills the same rate whether it is training a model or sitting idle behind a crashed process or forgotten job, so a week of low utilization on an expensive fleet can quietly cost more than a full outage that gets fixed fast.
In distributed training, the all-reduce synchronization step runs at the speed of the slowest rank, so a single degrading GPU, even one with just a climbing ECC error rate, can make every other card in the job look equally slow.
NVIDIA's Data Center GPU Manager, or DCGM, is the telemetry layer most GPU health signals like utilization, ECC counts, and thermal state are collected from. It reports per-metric symptoms, not the correlation across layers needed to find a root cause.
Mean Time to Recovery on a GPU fleet measures how fast a stalled or crashed job gets back to useful work. Because every minute of GPU time is billed at the most expensive compute rate in the datacenter, slow diagnosis costs far more than on a typical web service.
Further Reading
Why Incident Debugging Is Still Slow in 2026
The 7 reasons observability does not equal understanding, and what actually needs to change.
What Is AI SRE in 2026
How AI is transforming site reliability engineering and what it means for incident response teams.
The Four Pillars of Telemetry
Metrics, logs, traces, and events: what each measures and how they work together.
Why AI Agents Fail in Production
The broader pattern behind why AI agents fail silently, and what to do about it.
Never Miss What's Breaking in Prod
Breaking Prod is a weekly newsletter for SRE and DevOps engineers.
Subscribe on LinkedIn →