NVIDIA says automated recovery can reduce training downtime

NVIDIA said on Nov. 2, 2026, that distributed AI training increasingly needs automated recovery as failures across hardware and software dependencies can stall progress, idle GPUs and raise compute costs. The company framed NVIDIA Recovery, part of NVIDIA DSX OS, as an open-source, workload-level service designed to detect interruptions and coordinate restarts across a cluster’s training jobs.

What NVIDIA said Recovery does

In the blog post, NVIDIA said Recovery uses application logs, GPU metrics and scheduler signals to detect crashes, hangs and stragglers, then applies failure attribution to identify implicated resources and coordinate isolation and checkpoint restarts on healthy capacity. The company said the system is intended to automate recovery across workloads around the clock, reducing manual intervention and helping sustain training goodput.

The post said Recovery sits at the cluster level, distinct from infrastructure remediation and from in-job libraries such as NVRx. NVIDIA described break-fix automation as handling faulty nodes, while NVRx provides recovery, hang detection and straggler mitigation within individual jobs. Recovery, by contrast, is meant to manage cross-workload restarts, placement and node exclusion.

Integration with Slurm and the resilience stack

NVIDIA said the service integrates with Slurm and can work alongside other components in its health and resilience stack. The post said Kubernetes deployments may use NVSentinel to monitor GPU health metrics and XID events and automatically cordon and drain faulty nodes, while many Slurm sites rely on custom prolog and epilog scripts. NVIDIA said the slurm-operator from the Slinky project natively supports and injects those scripts when running Slurm on Kubernetes.

The company added that these layers operate independently without hard dependencies, so users can adopt them modularly or plug in their own platform components. It said NVIDIA Recovery can fit into existing health workflows with little or no change where NVSentinel or NVIDIA Health Validation is already in place.

Why the company says the gap matters

NVIDIA cited a simple reliability example: if each GPU operates without failure on a given day with 99.9% probability, a job spanning 10,000 GPUs completes that day uninterrupted with a probability of only 0.999^10,000, or about 0.005%. The company used that illustration to argue that rare per-GPU failures compound at cluster scale.

It also pointed to an 11-month study of 24,000 A100 GPUs at Meta, saying hardware failures interrupted fewer than 1% of jobs while those jobs accounted for 19% of GPU runtime. NVIDIA said the largest and longest-running jobs are the most exposed to interruption.

Operational data from one training cluster

NVIDIA said its production observations from July 9 to Oct. 9 showed monitoring on one training cluster covering nearly 200 jobs per day and recording approximately 8,550 anomaly detections. The company said mean detection latency was 69 seconds and mean recovery-action latency was 94 seconds. It defined recovery-action latency as ending at the automated action, adding that scheduling and checkpoint restoration determine when productive training resumes.

The post did not specify the cluster’s size, the nature of the jobs, or how much productive training time was restored. It also did not quantify performance gains from Recovery relative to manual processes.

Automated recovery loop

NVIDIA said the recovery loop has four stages: detect, attribute, isolate and resume. In the detect stage, the Anomaly Detector combines application logs and scheduler signals to identify crashes, hangs and stragglers. The article was truncated before detailing the remaining stages in full, but it said the system’s purpose is to coordinate recovery actions across workloads rather than only within a single job.