AI Operations
Free NVIDIA-Certified Associate: AI Infrastructure and Operations practice — 6 questions on AI Operations, with explanations. No sign-up. Full 12-question mixed test →
An AI training cluster running multi-node PyTorch distributed training jobs on DGX systems experiences intermittent NCCL timeout errors. DCGM shows GPU utilization drops to 0% on some nodes during these failures, while nvidia-smi reports no GPU errors. Network bandwidth monitoring shows normal IB traffic. Which troubleshooting step should be performed FIRST?
NCCL timeouts with zero GPU utilization during multi-node training typically indicate communication path issues rather than compute problems. Verifying NVLink/NVSwitch topology ensures GPU-to-GPU communication paths are correctly established. Misconfigured or degraded NVLink connections cause NCCL collective operations to fail or fall back to slower paths, triggering timeouts. This is the most direct diagnostic for inter-GPU communication issues in DGX systems.
A production inference cluster managed by NVIDIA Base Command Platform needs to support both high-priority real-time inference requests (P0) and lower-priority batch scoring jobs (P1). The cluster has 8 A100 GPUs. Which resource allocation strategy BEST ensures P0 workloads always have immediate GPU access while maximizing overall cluster utilization?
Kubernetes resource quotas with preemption provide the optimal balance: guaranteeing capacity for high-priority workloads while allowing low-priority jobs to use idle resources. Preemption enables immediate eviction of P1 pods when P0 workloads arrive, ensuring zero wait time for critical requests. This approach maximizes utilization by allowing P1 to consume unused capacity rather than leaving GPUs idle. Base Command Platform supports this Kubernetes-native priority and preemption model.
DCGM metrics show that GPUs in a training cluster have consistent 95%+ SM (streaming multiprocessor) utilization but only 60% memory bandwidth utilization. Training throughput is 40% below expected performance. Which optimization should be implemented?
High SM utilization with low memory bandwidth utilization indicates the GPUs are compute-bound but not effectively using memory subsystem capacity. This pattern suggests small batch sizes or inefficient memory access patterns. Increasing batch size improves memory bandwidth utilization by creating larger, more efficient memory transfers and better amortizing fixed costs of memory operations. With memory bandwidth underutilized at 60%, there is headroom to increase batch size without hitting memory limits.
A Slurm cluster running AI training workloads needs to enforce fair-share scheduling across three research teams with different priorities: Team A (weight 50), Team B (weight 30), Team C (weight 20). Team A has submitted jobs consuming all available GPUs for the past week. Team B now submits urgent jobs but they remain queued. Which Slurm configuration change will allow Team B jobs to start while maintaining fair-share principles?
Fair-share scheduling in Slurm uses PriorityDecayHalfLife to decay historical usage over time, ensuring teams that have consumed significant resources see their priority decrease while underutilized teams gain priority. Setting a 7-day half-life with priority/multifactor enables the scheduler to consider Team A's week-long GPU consumption, lower their priority, and boost Team B's priority based on their underutilization relative to their 30% fair-share target. This implements true fair-share principles dynamically.
An MLOps pipeline deploys inference models to production using NVIDIA Triton Inference Server on Kubernetes. After deploying a new model version, p99 latency increases from 15ms to 45ms despite GPU utilization remaining at 30%. DCGM shows no thermal throttling and nvidia-smi reports normal clock speeds. Which diagnostic step will MOST likely identify the root cause?
A 3x increase in p99 latency with low GPU utilization suggests the model is not executing efficiently on the GPU, possibly due to misconfiguration. Triton server logs would reveal if the model backend is mismatched (e.g., ONNX model being run on PyTorch backend), causing fallback to CPU execution or inefficient GPU code paths. Model format mismatches are common deployment errors that cause severe performance degradation while showing low GPU utilization because the GPU is waiting or the work is being done on CPU. This is the most direct diagnostic for sudden post-deployment latency spikes.
A GPU cluster experiences recurring ECC (Error Correction Code) memory errors on specific A100 GPUs. DCGM reports 5-10 correctable ECC errors per hour on 2 out of 16 GPUs, with no uncorrectable errors. Training jobs complete successfully but occasionally show NaN losses on these GPUs. What is the BEST operational response?
Correctable ECC errors at 5-10/hour indicate degrading memory that will likely progress to uncorrectable errors. While correctable errors are automatically fixed by hardware, their increasing frequency suggests the memory cells are failing. The correlation with NaN losses indicates some errors are affecting computation before correction. Best practice is to schedule maintenance replacement (to avoid sudden failures) while immediately preventing new critical workloads from being scheduled (using taints). This balances operational continuity with proactive hardware management, preventing data corruption and job failures.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →