Serving and scaling models
Free Google Cloud Professional Machine Learning Engineer practice — 6 questions on Serving and scaling models, with explanations. No sign-up. Full 12-question mixed test →
A team deploys 40 low-traffic fraud-detection models, one per merchant, each needing its own Vertex AI endpoint for separate IAM and monitoring. Deploying each model to a dedicated n1-standard-4 machine type is costing far more than expected because most models receive only a few requests per hour. What should they do to reduce compute cost while keeping the models on separate endpoints?
Deployment Resource Pools let multiple deployed models (even across different endpoints) share the same set of VM/accelerator resources, which is exactly the pattern for many low-QPS models that don't each need dedicated nodes -- the endpoints and IAM stay separate, but compute is pooled.
A healthcare company must call its Vertex AI online prediction endpoint from an on-premises data center connected via Cloud Interconnect, and network security requires that prediction traffic never traverse the public internet or receive a publicly routable IP. Which Vertex AI serving configuration satisfies this requirement?
Vertex AI private endpoints exposed via Private Service Connect place prediction traffic entirely on private Google network paths reachable from a peered/connected VPC, with no public IP involved -- this is the supported pattern for on-prem-to-Vertex AI serving over Interconnect without touching the public internet.
A team deploys a model on Vertex AI Prediction using n1-standard-8 nodes each with one NVIDIA T4 GPU attached. They configure autoscaling based only on CPU utilization with a 60% target, but during load tests the endpoint fails to scale up even though GPU-bound inference latency spikes badly under load. What is the most likely cause and fix?
When a deployment uses accelerators, CPU utilization is often a poor proxy for load because the GPU is the actual bottleneck; Vertex AI supports autoscaling on the accelerator duty cycle metric (aiplatform.googleapis.com/prediction/online/accelerator/duty_cycle), which is the correct signal to configure for GPU-backed endpoints.
A retailer has Model V1 serving 100% of traffic on a Vertex AI endpoint. They want to gradually roll out Model V2, sending it only 5% of live traffic initially so they can compare business KPIs before a full cutover, without redeploying V1. What is the correct way to do this on Vertex AI?
Vertex AI endpoints support deploying multiple models simultaneously and setting a traffic_split map (e.g., {v1_id: 95, v2_id: 5}) so a percentage of live prediction requests is routed to each deployed model automatically -- this is the native mechanism for canary/A-B rollout without any client-side logic or redeployment.
An image classification model deployed on a Vertex AI endpoint with an NVIDIA T4 GPU must respond in under 50ms at the 95th percentile for a mobile app, but current p95 latency is 120ms even after increasing max_replica_count. Profiling shows the GPU itself, not queuing, is the bottleneck per-request. What should the team try next to reduce per-request latency without changing the model's architecture?
Since queuing was ruled out and the bottleneck is per-request GPU compute time, the fix must reduce the actual inference compute cost. Converting to an optimized inference runtime (e.g., TensorRT) with INT8 quantization reduces per-request latency on the same GPU by using faster kernels and lower-precision math, directly addressing a compute-bound bottleneck.
A logistics company needs to score 200 million shipment records nightly with a trained tabular model, writing predictions back to BigQuery. There is no requirement for sub-second responses, and the pipeline must run unattended on a schedule with minimal ongoing infrastructure cost. Which serving approach best fits these requirements?
Vertex AI batch prediction is purpose-built for large-scale, non-latency-sensitive scoring jobs, natively supports BigQuery as both input and output, runs on-demand without keeping infrastructure always on, and is the most cost-efficient option for a scheduled nightly job over 200M rows.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$129.99 $34.99 with code FREETEST33 — valid through September 7.
Get my $34.99 deal →