AI Infrastructure
Free NVIDIA-Certified Associate: AI Infrastructure and Operations practice — 6 questions on AI Infrastructure, with explanations. No sign-up. Full 12-question mixed test →
A financial services company is deploying a 64-GPU cluster for LLM training with strict power budget constraints. The data center can provide 120kW per rack. Each DGX H100 system consumes approximately 10.2kW at peak load. The infrastructure team needs to determine rack density while maintaining N+1 redundancy for power distribution. What is the MAXIMUM number of DGX H100 systems that can safely be deployed per rack?
With 120kW available and 10.2kW per system, raw math allows ~11 systems. However, N+1 redundancy requires planning for one power supply failure per system, meaning each system needs dual feeds from separate PDUs. With redundancy overhead (typically 40-50% for N+1), safe deployment is 5 systems per rack (51kW base + redundancy = ~102kW total).
An AI research lab running multi-node training jobs across 16 DGX A100 systems is experiencing inconsistent training performance. Network monitoring shows periodic 40% throughput drops during gradient synchronization phases. The cluster uses 200GbE Ethernet with RoCEv2. Which configuration change would BEST address this performance degradation?
InfiniBand with GPU Direct RDMA eliminates CPU involvement in GPU-to-GPU communication, providing deterministic low-latency communication essential for multi-node training. HDR InfiniBand (200Gbps) with GPU Direct RDMA reduces gradient synchronization overhead by 60-80% compared to RoCEv2 on Ethernet. This directly addresses the collective communication bottleneck during all-reduce operations.
A cloud service provider needs to offer AI inference services with strict SLA guarantees for latency-sensitive applications while maximizing GPU utilization across diverse workloads. Some customers run BERT inference (small batch, low latency required), others run Stable Diffusion (larger batch, moderate latency acceptable). The infrastructure uses A100 80GB GPUs. Which multi-tenancy strategy provides the BEST balance of isolation and utilization?
MIG (Multi-Instance GPU) provides hardware-level isolation with guaranteed QoS—each instance has dedicated memory and SM resources. 1g.10gb instances (1/7 GPU, 10GB) are optimal for BERT inference (small models, low latency needs), while 3g.40gb instances (3/7 GPU, 40GB) suit Stable Diffusion workloads. MIG ensures no noisy neighbor issues and deterministic latency, critical for SLA compliance.
A pharmaceutical company is sizing storage infrastructure for a drug discovery pipeline that will train foundation models on 500TB of molecular structure data. Training jobs perform random access to 15% of the dataset per epoch with 8 DGX H100 nodes reading concurrently. The storage system must sustain 800GB/s aggregate read throughput during training. Which storage architecture BEST meets these requirements?
Parallel filesystems like WEKA or BeeGFS stripe data across multiple NVMe-backed storage servers, enabling aggregate throughput to scale linearly with nodes. Combined with 200GbE RDMA (GPU Direct Storage capable), this architecture can deliver 800GB/s+ aggregate throughput. Random access patterns require NVMe (not HDD), and distributed architecture prevents single-point bottlenecks.
An enterprise ML platform team is evaluating GPU deployment options for a hybrid infrastructure supporting both on-premise DGX systems and cloud-based training. They need to run identical containerized training workloads across both environments with minimal portability friction. Training jobs use NCCL for multi-GPU communication and require consistent performance profiling. Which combination of technologies provides the MOST seamless hybrid deployment experience?
NVIDIA Container Toolkit enables GPU access in Docker containers on-prem and in cloud VMs identically. Kubernetes provides consistent orchestration (pod specs, resource requests, scheduling policies work identically). NGC registry stores GPU-optimized containers that run unchanged across environments. This stack is NVIDIA's recommended approach for hybrid AI infrastructure.
A university research cluster with 8 DGX A100 nodes connected via NVSwitch is experiencing suboptimal scaling efficiency. Single-node training achieves 95% GPU utilization, but 8-node distributed training drops to 68% efficiency during all-reduce operations. Network topology uses a single 200GbE switch with NCCL configured for ring algorithm. Profiling shows 45% of iteration time spent in NCCL calls. What is the PRIMARY bottleneck?
8 DGX A100 nodes with 8 GPUs each = 64 GPUs total. During all-reduce, each GPU must exchange gradients with all others. Ring all-reduce requires bandwidth proportional to (N-1)/N × data × GPUs. With 200GbE (25GB/s) shared across 8 nodes, per-node bandwidth is ~3.1GB/s, but each DGX can generate 600GB/s+ intra-node via NVSwitch. The network is a 95x bottleneck, explaining 45% iteration time in NCCL.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →