Evaluate and deploy agentic workflows
Free Google Cloud Professional Agentic Architect practice — 6 questions on Evaluate and deploy agentic workflows, with explanations. No sign-up. Full 12-question mixed test →
You are evaluating a customer-service agent using the Vertex AI Gen AI evaluation service's trajectory metrics. The business rule requires that check_inventory must be called before create_order, but it is acceptable for the agent to also call helper tools like log_event or fetch_user_profile at any point in between or before those two calls. Which trajectory metric should you use to score predicted trajectories against the reference trajectories?
in_order_match checks that the reference actions appear in the predicted trajectory in the correct relative order while still tolerating additional (extra) actions interspersed elsewhere.
Your team wants to add an automated quality gate to the Cloud Build pipeline that blocks promotion of a new agent version to production if its groundedness and tool-use accuracy regress compared to the currently deployed version. Which evaluation approach should the pipeline step implement?
Offline evaluation runs the candidate agent against a fixed, version-controlled reference dataset via the Gen AI evaluation service, producing deterministic groundedness and tool-use scores that can be compared release-over-release and used to block promotion before any production traffic is affected.
A fintech company deploys its ADK-based agent to Vertex AI Agent Engine. For the next release, they want the ability to instantly roll back to the previous agent behavior if the new release causes a spike in hallucinated financial advice, with zero downtime. Which deployment strategy should they use?
Deploying the new build as a separate, immutable Agent Engine resource (blue-green/canary) lets you gradually shift traffic and instantly revert by routing clients back to the still-running prior resource, with no in-place mutation and no downtime.
Users report that a deployed agent occasionally takes over 8 seconds to respond, but Cloud Logging shows only the final response text with no indication of where the time is spent. What should you do to pinpoint whether the delay comes from the LLM call, a specific tool invocation, or a retrieval step?
Cloud Trace integration for Agent Engine/ADK captures a distributed trace with individual spans for each LLM call, tool call, and retrieval step, letting you see exactly which span consumed the 8 seconds.
Your agent must meet an SLA of p95 latency under 900ms for a high-volume support chatbot while keeping per-request cost low; the reasoning required is simple intent classification plus a single tool call. Which model and configuration choice BEST meets this SLA and budget?
For simple intent classification and a single tool call, Gemini Flash's lower latency and lower per-token cost, combined with a minimal prompt and disabled/low thinking budget, directly targets the sub-900ms p95 and low-cost requirements without unnecessary reasoning overhead.
You want a CI/CD pipeline for an agent that (1) blocks release if automated evaluation detects a drop in groundedness or an increase in tool-misuse rate, and (2) still catches rare failure modes that only emerge under real user traffic, with minimal risk to all users. Which pipeline design BEST satisfies both goals?
Combining a pre-deploy offline evaluation gate (catches known regressions against reference data) with a canary release monitored by online evaluation and automated rollback minimizes blast radius while satisfying both the regression-blocking and rare-failure-detection goals.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →