Deployment, Serving, and Monitoring
Free NVIDIA-Certified Professional: Generative AI and LLMs practice — 6 questions on Deployment, Serving, and Monitoring, with explanations. No sign-up. Full 12-question mixed test →
A team serves an LLM via Triton Inference Server behind a strict 50ms p99 latency SLA. Throughput is inadequate at low concurrency even though GPU utilization is low. Which config.pbtxt change BEST increases throughput while still respecting the SLA?
Dynamic batching with a preferred_batch_size matched to real traffic and a small max_queue_delay_microseconds fills batches for throughput gains without exceeding the millisecond-scale latency budget left after the SLA.
A financial services company must replace a production LLM (v1) with a newly fine-tuned version (v2) with zero downtime, and needs the ability to instantly roll back if the hallucination rate exceeds a threshold once observed on real user traffic. Which deployment pattern BEST satisfies these requirements?
Canary rollout exposes only a small fraction of live users to v2, lets real-traffic quality metrics (like hallucination rate) be measured against a defined threshold, and supports automated, near-instant rollback with zero downtime.
A platform must serve 200 customer-specific LoRA-adapted variants of the same base LLM cost-effectively, with sub-second adapter switching and without loading 200 full model copies onto GPUs. Which approach BEST meets this on the NVIDIA stack?
NIM's multi-LoRA support serves one shared base model on GPU and hot-swaps lightweight adapter weights per request, minimizing memory footprint and enabling fast, dynamic switching across many tenants.
In Triton Inference Server, what is the key functional difference between 'dynamic batching' and 'concurrent model execution' (instance groups)?
Dynamic batching groups independent requests into a batch for a single execution to raise throughput; instance groups configure how many parallel copies of the model run to serve concurrent requests independently — the two are complementary but distinct mechanisms.
A production RAG-based LLM system's monitoring dashboard shows a gradual increase in retrieval-answer mismatch rate over three weeks, with no code, model, or infrastructure changes deployed in that period. What is the MOST likely root cause and appropriate MLOps action?
A slow, steady degradation in answer quality with no infra or code change is the textbook signature of data or concept drift — the corpus or query patterns have shifted away from what the retriever/model were tuned on, warranting re-indexing and a retraining/re-tuning evaluation per standard LLM lifecycle practice.
An LLM inference service running Triton on Kubernetes experiences bursty traffic. The team configured the Horizontal Pod Autoscaler (HPA) using average CPU utilization, but users still see latency spikes during bursts even though CPU stays below 40%. What is the BEST fix?
LLM inference load is GPU-bound, so CPU utilization is a poor autoscaling signal; scaling on GPU utilization or Triton's queue depth (via a custom metrics adapter or KEDA) accurately reflects real inference pressure and triggers timely scale-out during bursts.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$129.99 $34.99 with code FREETEST33 — valid through September 7.
Get my $34.99 deal →