Inference Concepts
Free NVIDIA-Certified Associate: AI Model & Application Development practice — 6 questions on Inference Concepts, with explanations. No sign-up. Full 12-question mixed test →
A team deploys an LLM chatbot that supports long conversation histories (8K+ tokens). Despite using a small batch size, the GPU repeatedly runs out of memory mid-conversation as context grows. Which optimization BEST addresses this specific problem?
KV cache grows linearly with sequence length and can fragment GPU memory over long contexts. Paged/chunked KV cache management (e.g., PagedAttention-style allocation used in TensorRT-LLM) allocates memory in fixed-size blocks on demand, eliminating fragmentation and OOM errors for long-context workloads.
An inference service receives a continuous stream of LLM requests with highly variable output lengths. The team wants to maximize GPU utilization and throughput without forcing new requests to wait for an entire batch to finish before starting. Which technique BEST fits this requirement?
Continuous (in-flight) batching dynamically admits new requests and removes finished sequences on a per-iteration basis, which is specifically designed for variable-length LLM generation and maximizes GPU utilization without the idle time static batching introduces.
In Triton Inference Server's model configuration file, which setting enables request batching with a configurable wait window for incoming inference requests?
The dynamic_batching block in a Triton model config enables automatic batching of incoming requests, and max_queue_delay_microseconds controls how long Triton waits to form a larger batch before dispatching, directly trading a small latency increase for higher throughput.
Which statement correctly distinguishes latency optimization from throughput optimization in model inference serving?
Latency and throughput are often in tension: larger batches increase GPU utilization and total throughput but add queuing delay that raises per-request latency. Understanding this tradeoff is core to configuring inference servers for production SLAs.
A team must deploy an LLM on an edge device with severely limited VRAM. A few percentage points of accuracy loss are acceptable in exchange for the smallest possible memory footprint and fastest inference. Which precision/quantization choice BEST fits these constraints?
INT4 quantization compresses weights to 4 bits, giving the smallest memory footprint and fastest inference among common options, at the cost of some accuracy — exactly matching a severely VRAM-constrained edge scenario where a few percent accuracy loss is acceptable.
A development team needs to deploy a pre-optimized, production-ready LLM with a standardized REST/gRPC API, running either on-prem or in the cloud, with minimal custom serving infrastructure to build. Which NVIDIA offering BEST fits this requirement?
NVIDIA NIM (NVIDIA Inference Microservices) packages pre-optimized models (using TensorRT-LLM/Triton under the hood) with standardized APIs in containers, letting teams deploy production inference endpoints in minutes without building custom serving infrastructure.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →