Workload Management
Free NVIDIA AI Operations Professional (NCP-AIO) practice — 6 questions on Workload Management, with explanations. No sign-up. Full 12-question mixed test →
A data science team reports that their multi-node NCCL training job running across 4 DGX nodes consistently hangs during the allreduce operation after 2-3 epochs. You run 'dcgmi diag -r 3' and observe intermittent PCIe link width degradation on GPU 2 of node dgx-03. The Slurm job was submitted with '--ntasks-per-node=8 --gpus-per-task=1'. Which action will MOST effectively resolve this issue?
PCIe link width degradation detected by DCGM diagnostics indicates a hardware issue that will cause unreliable GPU communication. The correct procedure is to drain the node (removing it from scheduling while preserving running jobs), physically address the hardware problem (reseating GPU or checking riser card connection), verify with diagnostics, then return to production. Running jobs on degraded hardware leads to data corruption and training failures.
Your organization uses Run:ai to manage GPU resources across three departments. The ML Research team (department-a) has a guaranteed quota of 16 GPUs but frequently submits jobs requesting 32 GPUs during off-peak hours. The Platform team wants to allow over-quota usage when resources are idle but ensure department-b's guaranteed 8 GPUs are immediately available when needed. Which Run:ai configuration achieves this?
Run:ai's deserved-gpus defines guaranteed allocation, and over-quota-priority determines which department's over-quota jobs get preempted first when guaranteed resources are needed. Setting department-a's over-quota-priority=1 (lower priority) and department-b's to 0 (higher) ensures department-a's over-quota jobs are preempted when department-b needs its guaranteed 8 GPUs. Enabling preemption on department-a's project allows Run:ai to automatically reclaim resources.
You are deploying Triton Inference Server on Kubernetes to serve 3 different models: model-a (ONNX, 2GB), model-b (TensorRT, 8GB), and model-c (PyTorch, 12GB). The cluster has nodes with A100-40GB and A100-80GB GPUs. Model-c requires CUDA 12.2 features, while model-a and model-b run on CUDA 11.8. To minimize GPU memory waste while ensuring all models have dedicated GPU access, which deployment strategy is BEST?
Model-a (2GB) + model-b (8GB) = 10GB total, fitting comfortably on one A100-40GB with CUDA 11.8. Model-c (12GB) requires CUDA 12.2 and fits on A100-40GB but using A100-80GB provides headroom for concurrent requests. This approach minimizes waste: one A100-40GB serves two models (30GB free for batching/overhead), one A100-80GB serves model-c (68GB free for large batches). Separating by CUDA version prevents container conflicts.
A researcher submits a Slurm job using 'sbatch --partition=gpu --gres=gpu:4 --nodes=2 train.sh' that launches a PyTorch DistributedDataParallel training job. The job starts but only utilizes GPUs on the first allocated node. Checking the Slurm output shows both nodes allocated (node-[05-06]), and 'srun --jobid=12345 hostname' returns both hostnames. The train.sh script uses 'torchrun --nproc_per_node=4 train.py'. What is the MOST likely cause?
In Slurm, torchrun (or python -m torch.distributed.launch) runs only on the node where the script executes unless wrapped with srun. The correct pattern for multi-node PyTorch DDP with Slurm is 'srun torchrun --nproc_per_node=4 train.py' or use srun with --ntasks-per-node and torch.distributed.launch. Without srun, train.sh runs on the first allocated node only, and torchrun spawns 4 local processes there, leaving node-06's GPUs idle. Exam tests this Slurm+PyTorch integration pattern.
Your team is deploying NVIDIA NIM microservices for Llama-2-70B from NGC on a Kubernetes cluster. After pulling the NIM container and creating a deployment with 'resources.limits.nvidia.com/gpu: 2', the pod remains in Pending state. Running 'kubectl describe pod' shows 'insufficient nvidia.com/gpu'. You verify 4 A100-80GB GPUs are available on the node using 'nvidia-smi'. Which configuration issue is MOST likely preventing the pod from scheduling?
Kubernetes does not natively understand GPUs as schedulable resources. The NVIDIA device plugin daemonset runs on each GPU node, detects GPUs via nvidia-smi, and advertises them to the kubelet as 'nvidia.com/gpu' extended resources. Without the device plugin running, GPUs are invisible to the scheduler even though they exist physically. The 'insufficient nvidia.com/gpu' error specifically indicates the resource type is not available in the cluster's allocatable resources. Exam tests understanding that device plugin is mandatory infrastructure.
You need to run a multi-node benchmark using NCCL tests (nccl-tests/build/all_reduce_perf) across 8 DGX A100 nodes with 8 GPUs each to validate InfiniBand fabric performance before deploying production training workloads. The Slurm cluster uses 'gpu' partition with nodes dgx-[01-16]. Which sbatch script configuration will correctly execute the all_reduce_perf test using all 64 GPUs with proper MPI and NCCL settings?
NCCL tests use MPI for multi-node process launching but NCCL for GPU communication. The correct pattern: --nodes=8 (8 nodes), --ntasks-per-node=8 (one MPI rank per GPU), --gpus-per-task=1 (each rank gets exclusive GPU access), --exclusive (no sharing). Using 'srun --mpi=pmix' leverages Slurm's native MPI support (pmix is modern default). The -g 1 flag tells all_reduce_perf each rank uses 1 GPU. This creates 64 MPI ranks (8 nodes × 8 tasks), each with one GPU, which is the standard pattern for NCCL multi-node benchmarks.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$199.99 $34.99 with code FREETEST33 — valid through September 2.
Get my $34.99 deal →