Optimization and Acceleration
Free NVIDIA-Certified Professional: Generative AI and LLMs practice — 6 questions on Optimization and Acceleration, with explanations. No sign-up. Full 12-question mixed test →
A team trains a 70B-parameter model on 64 GPUs arranged as 8 nodes of 8 GPUs each, with NVLink connecting GPUs within a node and InfiniBand connecting nodes. The model does not fit on a single GPU due to activation memory even after optimizer-state sharding. Which parallelism mapping minimizes cross-node communication overhead while still fitting the model in memory?
Tensor parallelism requires frequent, latency-sensitive all-reduce operations best suited to the high-bandwidth, low-latency NVLink within a node. Pipeline parallelism has lower communication volume (only activations/gradients at stage boundaries) and tolerates the higher latency of InfiniBand across nodes, giving an efficient 3D-parallel mapping.
An engineer needs to reduce GPU memory consumed by optimizer states and gradients during training but wants to avoid the parameter all-gather communication overhead that occurs during the forward pass. Which ZeRO stage should be selected?
ZeRO Stage 2 shards both optimizer states and gradients across data-parallel ranks, delivering substantial memory savings while keeping full parameters replicated locally, so no parameter all-gather is needed during the forward pass.
When enabling FP8 mixed-precision training on H100 GPUs using NVIDIA Transformer Engine, which components are typically kept in FP8 and which remain in higher precision to preserve model accuracy?
Transformer Engine applies FP8 with per-tensor dynamic scaling specifically to the compute-heavy GEMM operations in linear layers, since these tolerate reduced precision well, while numerically sensitive operations (layer norm, softmax) and the master copy of weights are kept in BF16/FP32 to maintain training stability and convergence.
A team implements speculative decoding for a 70B-parameter model to reduce per-token latency using a smaller draft model. During testing, only 20% of speculated tokens are accepted, and overall latency INCREASES compared to standard autoregressive decoding. What is the most likely cause?
With only a 20% acceptance rate, most speculated tokens are rejected and must be recomputed by the target model anyway. If the draft model itself is not cheap/fast enough relative to this low acceptance rate, the added draft-model inference cost plus rejection-driven recomputation can exceed the latency of plain autoregressive decoding, producing a net slowdown.
A team needs to quantize a large language model's weights to 4 bits for deployment on memory-constrained GPUs. The model exhibits significant activation outliers in certain channels, and accuracy degradation must be minimized without an extensive calibration process. Which quantization approach is BEST suited?
AWQ identifies and protects the small subset of weight channels that correspond to large-magnitude activations by scaling them before quantization, preserving accuracy at 4-bit precision with a lightweight calibration process, making it well suited to models with pronounced activation outliers.
Gradient checkpointing (activation recomputation) reduces GPU memory usage during training by discarding intermediate activations and recomputing them during the backward pass. What is the primary tradeoff of enabling this technique?
Gradient checkpointing trades compute for memory: activations are recomputed via an extra forward pass through the checkpointed segments during backpropagation, which typically adds roughly 20-30% additional training time in exchange for substantial memory savings.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$129.99 $34.99 with code FREETEST33 — valid through September 7.
Get my $34.99 deal →