Troubleshooting and Optimization
Free NVIDIA AI Operations Professional (NCP-AIO) practice — 6 questions on Troubleshooting and Optimization, with explanations. No sign-up. Full 12-question mixed test →
A multi-node DGX A100 cluster is experiencing severe performance degradation during large-scale distributed training. NCCL benchmarks show expected bandwidth on intra-node communication but 10x slower than expected bandwidth on inter-node AllReduce operations. The fabric manager service is running on all nodes. Inspecting `/var/log/fabricmanager.log` reveals repeated "NVSwitch port flapping detected on switch 0, port 12" messages on multiple nodes. Which action will MOST effectively resolve the inter-node communication bottleneck?
Port flapping on NVSwitch indicates physical layer issues with NVLink cable connections. Reseating cables addresses the root cause (loose or degraded cable connections), and restarting fabric manager after physical repair allows proper re-enumeration of the NVLink fabric topology. This directly resolves the inter-node bandwidth degradation.
An AI training job repeatedly fails when pulling the `nvcr.io/nvidia/pytorch:24.03-py3` container image with the error "error pulling image configuration: unknown blob". The NGC API key is correctly configured in `/etc/nvcr.io/config.json`. Running `docker pull nvcr.io/nvidia/cuda:12.3.1-base-ubuntu22.04` succeeds without errors. Which single action is the BEST next troubleshooting step to isolate the root cause?
The error 'unknown blob' during configuration pull, combined with successful pulls of other images from the same registry, indicates a problem specific to this image's manifest or layers. Docker daemon logs will reveal whether the issue is a corrupted manifest, a specific layer blob unavailable on the registry, or a transient registry error for this particular image. This provides diagnostic data to determine if the problem is registry-side or client-side.
A Base Command Manager cluster has 8 DGX nodes, but only 6 appear in the BCM UI node inventory. SSH access to all 8 nodes works correctly. Running `bcm-agent status` on the two missing nodes shows "bcm-agent is not running". Checking `/var/log/bcm-agent.log` on one missing node shows "Certificate validation failed: unable to get local issuer certificate" repeated every 30 seconds. What is the MOST likely cause and correct resolution?
The error 'unable to get local issuer certificate' during TLS validation means the node does not trust the BCM manager's certificate authority. BCM uses internal PKI; nodes must have the cluster CA certificate in their trusted store. Copying the CA bundle from a working node and updating the system certificate store resolves the trust chain, allowing bcm-agent to establish secure connections to the manager.
A containerized training workload using GPUDirect Storage (GDS) to read training data from a local NVMe RAID array is achieving only 3 GB/s read throughput per GPU, while the storage array is capable of 28 GB/s aggregate. The container was launched with `--runtime=nvidia --gpus all -v /mnt/nvme:/data`. Running `cat /proc/driver/nvidia/capabilities/gds/status` inside the container shows "GDS Version: N/A". What must be added to the container launch configuration to enable GDS?
GPUDirect Storage requires access to the `/dev/nvidia-fs*` device nodes created by the nvidia-fs kernel driver. Containers do not inherit host device access by default; explicitly passing `--device=/dev/nvidia-fs0` (or all GDS devices with `--device=/dev/nvidia-fs0 --device=/dev/nvidia-fs1` etc.) exposes the GDS control plane to the container, enabling cuFile API to use direct GPU-to-storage I/O paths.
Multiple users report that training jobs running on a shared AI cluster experience intermittent 50% performance drops for 10-15 second periods. Monitoring shows GPU utilization dropping from 98% to 45% during these periods, while GPU memory usage remains constant. The jobs use a shared NFS-mounted dataset (`/nfs/datasets`) for training data. Running `nfsstat -c` during a performance drop shows the "retrans" counter increasing rapidly. Which configuration change will MOST effectively eliminate the performance drops?
Rapidly increasing retransmission counters indicate the single NFS server is overwhelmed by concurrent I/O requests from multiple training jobs. NFS is a single-server protocol that becomes a bottleneck with many clients. Parallel filesystems like BeeGFS or Lustre stripe data across multiple storage servers and provide aggregate bandwidth that scales with cluster size, eliminating the single-server bottleneck causing retransmissions and performance drops.
A new NGC container (`nvcr.io/nvidia/tensorflow:24.05-tf2-py3`) fails to start on a DGX A100 node with the error "Failed to initialize NVML: Driver/library version mismatch". The host is running NVIDIA driver version 535.104.05 and CUDA 12.2. Running `nvidia-smi` on the host works correctly and shows all 8 GPUs. Inspecting the container's build metadata shows it was built with CUDA 12.4. What is the correct action to resolve the container startup failure?
NVIDIA driver 535.104.05 supports CUDA 12.2 but not CUDA 12.4 (which requires driver ≥545.23.06). The container runtime uses the host's kernel driver; containers bundle CUDA libraries but rely on the host driver. NVML version mismatch occurs when container CUDA libraries expect newer driver features. Upgrading the host driver to ≥545.23.08 provides CUDA 12.4 API support, resolving the mismatch while maintaining backward compatibility with CUDA 12.2.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$199.99 $34.99 with code FREETEST33 — valid through September 2.
Get my $34.99 deal →