NVIDIA details Attribution tool for diagnosing distributed AI failures

NVIDIA has outlined a workload-attribution system designed to trace distributed AI failures back to their initiating event, rather than stopping at the downstream errors that appear across ranks and nodes. The company said the tool correlates logs, metrics, telemetry and scheduler state to produce a structured diagnosis that includes the outcome, root cause, affected nodes, supporting symptoms and an evidence trail.

How the system works

In a November 2, 2026 blog post, NVIDIA described Attribution as a workload-centric failure-attribution system that uses deterministic rules to make results reproducible and customizable. The company said the system is open source and is used across NVIDIA AI infrastructure. According to the post, it has analyzed millions of workloads each week across hundreds of thousands of GPUs.

The architecture consists of two deployable components: the NVIDIA Attribution Service and the NVIDIA Attribution Blueprint. The Service analyzes one workload attempt at a time through a REST API, while the Blueprint connects cluster schedulers and telemetry sources, gathers workload context and writes the result. NVIDIA said the process begins by scoping the workload, then collecting observations from application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history, and node and network metrics.

Example diagnosis

NVIDIA said a worked example shows how the system can distinguish a root cause from downstream symptoms in a multi-rank training job. In the example, one GPU encounters a hard fault, described as XID 149, and NCCL errors follow across other ranks. The system correlates those events, identifies the XID as the root cause, retains the NCCL failures in the symptom chain and names the responsible node.

The company also described a synthetic case in which application logs contained three different NCCL errors and system logs on node-02 showed XID 149 with error words that decode to an NVLink signal integrity issue. NVIDIA said the EARLY-stage filtering rule decodes the signal-integrity issue while leaving the NCCL events unchanged, and FINAL refinement surfaces the signal-integrity issue on node-02 while preserving the NCCL reports as interesting evidence.

Limits and uncertainty

NVIDIA said attribution quality depends on the quality and completeness of the underlying telemetry. When multiple causes fit the evidence, the company said the sub_domain is ambiguous. The post also said insufficient evidence can produce an unattributed result, indicating that the system is designed to reflect uncertainty rather than force a definitive diagnosis from incomplete data.

The article did not provide a launch date for broader availability beyond the November 2, 2026 post, and it did not disclose pricing or commercial terms.