Troubleshooting and Optimization
Free NVIDIA-Certified Professional: AI Infrastructure practice — 6 questions on Troubleshooting and Optimization, with explanations. No sign-up. Full 12-question mixed test →
During routine health checks on a DGX H100 node, `nvidia-smi -q -d XID` reports repeated Xid 79 events on GPU 3, and the GPU subsequently disappears from `nvidia-smi -L` output until the node is rebooted. What is the correct diagnostic and remediation path?
Xid 79 specifically means the GPU has dropped off the PCIe/NVLink bus, which is a hardware-level fault (power, thermal, seating, or board defect). It cannot be resolved by driver or software changes and requires physical inspection, reseating, or GPU replacement.
On an 8-GPU NVLink-connected node, `nvidia-smi nvlink --status` shows seven links reporting 'NVLink Speed: 25 GB/s' but one link between GPU0 and GPU4 reporting 'NVLink Speed: 6.25 GB/s'. All-reduce jobs using that GPU pair show measurable slowdowns. What should be checked next to isolate the cause?
A single NVLink degrading to a fraction of its rated speed while others operate normally points to a link-layer hardware fault. The correct isolation step is checking NVLink error/replay counters and system logs for Xid 74 (NVLink error) to confirm before escalating to hardware replacement.
A GPU cluster's multi-node NCCL all-reduce throughput measured with `all_reduce_perf` is 40% below the vendor-published expected bandwidth. Which command should be run FIRST to determine whether the bottleneck is in the physical InfiniBand fabric rather than the NCCL software stack?
ib_write_bw (perftest) measures raw RDMA throughput directly between HCAs, isolating the physical fabric from NCCL's collective algorithm and software overhead. If raw RDMA bandwidth is also degraded, the problem is in the network; if it's healthy, the issue is in the NCCL/software layer.
During sustained LLM training, `nvidia-smi -q -d PERFORMANCE` shows 'Clocks Throttle Reasons: SW Power Cap = Active' on all GPUs in a node, while GPU temperature stays 15°C below the thermal slowdown threshold. The rack's PDU still has 20% headroom before its circuit limit. Which action MOST directly resolves the throttling without violating power constraints?
'SW Power Cap' throttling occurs when the GPU hits its configured software power limit, not a thermal limit. Since temperatures are well within range and the rack still has power headroom, raising the power limit via nvidia-smi -pl is the direct fix that removes the artificial cap causing throttling.
Which NVIDIA Xid error code specifically indicates an NVLink error, as opposed to a GPU memory ECC or general driver fault?
Xid 74 is the code specifically documented as indicating an NVLink error, distinct from memory subsystem faults, and is the code an administrator should search logs for when link-level connectivity issues are suspected.
A distributed training job scales nearly linearly up to 4 nodes but throughput plateaus beyond 8 nodes. Per-GPU SM utilization remains at 95%+ throughout, and per-node network interface counters show links are not saturated. What is the MOST likely root cause to investigate first?
High per-GPU utilization with plateauing overall throughput and non-saturated individual links is a classic signature of collective communication inefficiency at scale — either a suboptimal NCCL algorithm (e.g., Tree vs Ring selection) or congestion at the spine/core switch layer during synchronized all-reduce bursts, which per-link average counters can mask.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$199.99 $34.99 with code FREETEST33 — valid through September 9.
Get my $34.99 deal →