NVIDIA AI Updates: November 3, 2026

1. NVIDIA Open Sources Attribution, Its Workload-Level Failure Diagnosis System for GPU Clusters

NVIDIA released NVIDIA Attribution as open source (github.com/dsx-ai-factory/attribution, Apache 2.0), a system it says has run in production across its own AI infrastructure for years and analyzes millions of workloads a week across hundreds of thousands of GPUs. It correlates workload-scoped application logs, syslogs, GPU XIDs, Fabric Manager messages, scheduler history and metrics through deterministic rules, then returns a structured verdict with terminal_state, an attributed sub_domain (marked ambiguous when several causes fit), the responsible nodes and the downstream symptoms, so an NVLink fault on one node is reported as the root cause rather than as a scatter of NCCL errors across ranks. It ships as two parts: an Attribution Service exposing the rules engine over HTTP, and a Python reference Blueprint with connectors for Slurm, Kueue, Loki, Kubernetes pod logs, local files and PromQL, writing to stdout or SQLite with a read-only dashboard. Rules are declarative extractors (plus Python rules) with inline test cases that the Service checks at startup; they are bundled with the Service rather than uploaded through the API, and the quickstart runs on sample evidence with Docker and uv. Source

2. NVIDIA Recovery Automates Detection, Node Exclusion and Checkpoint Restarts for Slurm Training Jobs

NVIDIA also open sourced NVIDIA Recovery (github.com/dsx-ai-factory/NVIDIA-Recovery, Apache 2.0), a cluster-level service that detects crashes, hangs and stragglers from application logs, GPU metrics and scheduler signals, excludes implicated nodes and restarts jobs from their last checkpoint or stops retries that are not making progress. It currently supports Slurm, including Slurm on Kubernetes via slurm-operator, through a reference Advisor Controller deployed by Helm in either in-cluster or SSH mode; workloads opt in with a single #SBATCH --comment directive, and native Kubernetes recovery for JobSet and Kubeflow PyTorchJob is on the roadmap. New deployments default to monitor mode, which only publishes recommendations until operators switch to action mode, and every decision lands in a persistent audit trail alongside the Attribution verdict when one is available. On one NVIDIA training cluster from July 9 to October 9, it covered nearly 200 jobs a day and logged about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds; IBM Research and the NVIDIA Cloud Partner Verda are early adopters, and NVIDIA describes the project as still working toward a stable release. Source

3. AI Cluster Runtime Adds a Preview Vera Rubin NVL72 Recipe, Validated at 744.71 GB/s NCCL All-Reduce

NVIDIA added a Vera Rubin NVL72 (vr200) recipe family to AI Cluster Runtime (AICR v1.0.0) for RKE2 on Ubuntu, with training, inference and Dynamo inference variants, released as Preview while the hardware is pre-release. The recipes encode what changed from GB200 NVL72: a host-installed R615.23 open driver (the tested version, not a confirmed minimum; early support is targeted for R610 and official support for R620), GPU Operator v26.7.0 with operator-managed drivers disabled, DRA driver v0.5.0 with IMEX ComputeDomains, Kubernetes 1.34.1 or later, Ubuntu 26.04 with a 64k-page arm64 kernel, ConnectX-9 allocation through DraNet, and a raised NFD topology updater memory limit because the 60Mi default gets OOM-killed on multi-terabyte nodes. On reference clusters, the training recipe passed all deployment and conformance checks plus a cross-node NCCL all-reduce gate over NVLink at 744.71 GB/s bus bandwidth with a 16 GiB message; the inference variant passed deployment and conformance, but inference serving performance, MIG and ConnectX-9 RDMA data-path performance are not yet validated. Teams doing fresh installs with GPU Operator v26.7.0 or later need to set numNodes explicitly in ComputeDomain manifests, because that chart ships a stricter CRD. Source

4. Cosmos Walkthrough Lists the Cosmos 3 Nano Simulation Checkpoints, GPU Memory Needs and the Cosmos3 2.0.0 NIM Container

NVIDIA published a hands-on guide to the Cosmos Framework and Cosmos NIM tools announced on October 22. The guide covers Cosmos3-Nano-Transfer-Auto (multi-view camera and LiDAR generation, up to 400 frames at 480p or 720p and 30 fps), its causal few-step variant Cosmos3-Nano-Sim-Auto for closed-loop AV policy evaluation, Sim-Transfer and Sim-Bimanual for robotics (YAM and AgiBot arms), and Cosmos3-H-Embodied-Nano and Cosmos3-H-Dreams for surgical simulation; they run on PyTorch and vLLM-Omni and are tested on Hopper and Blackwell, and the reference Transfer-Auto run uses four GPUs. The framework’s RL support now covers DPO, GRPO, PPO, on-policy distillation and DiffusionNFT. Cosmos3-Nano generation needs 58 GiB of VRAM in BF16, 44 GiB in FP8 and 31 GiB with FP8 plus offloading (Cosmos3-Super: 150, 93 and 38 GiB). The nvcr.io/nim/nvidia/cosmos3:2.0.0 container offers a latency profile on 1, 2, 4 or 8 GPUs and a throughput profile on 2, 4 or 8, and NVIDIA reports mean speculative-decoding speedups of roughly 2 to 3x on H200. Source

5. Workload Map Describes Kubernetes AI Workloads in YAML So Schedulers Skip Per-CRD Adapters

NVIDIA detailed Workload Map (formerly Karta; the CRD kind is still Karta), the Apache 2.0 CRD and Go library from the open-sourced Run:ai stack, which describes where a workload type keeps its pod templates, replica counts and status. Status mapping and scaling are written as jq expressions in YAML, and the catalog ships definitions for JobSet, PyTorchJob, RayCluster, RayJob, RayService, KServe, Knative, MPIJob, NVIDIA NIM, LeaderWorkerSet, Milvus, Grove PodCliqueSet and Dynamo. A kli CLI, installed from a Homebrew tap, renders a live workload as a role tree with GPU totals using only a kubeconfig. KAI Scheduler uses the definitions as a fallback for gang scheduling of types without a native plugin, Run:ai can register a new type with one API call, Elotl’s Nova uses them for multi-cluster placement, and NVIDIA Recovery plans to build on them for native Kubernetes support. Source