Evaluation, Testing, and Optimization
Free Claude Certified Architect - Professional practice — 6 questions on Evaluation, Testing, and Optimization, with explanations. No sign-up. Full 12-question mixed test →
An architect built an LLM-as-judge pipeline that performs pairwise comparisons between Model A's and Model B's outputs for a summarization task. Analysis shows the judge picks whichever response appears first in the prompt roughly 80% of the time, regardless of actual quality. What is the BEST way to correct this positional bias?
Swapping the order of the two candidate responses and averaging or reconciling the two resulting judgments is the standard documented technique for canceling out positional bias in pairwise LLM-as-judge setups, since it forces the bias to work against itself across the two trials.
A company runs 10,000 daily support-ticket classification requests. Each request includes an identical 3,000-token system prompt (containing instructions and examples) plus a unique 150-token ticket description. Results feed a dashboard refreshed once per day, so real-time responses are not required. Which approach BEST minimizes API cost?
Prompt caching and the Message Batches API discounts stack: caching the repeated 3,000-token system prompt yields up to a 90% discount on cached input tokens after the first write, while routing through Batches adds a further 50% discount on top of standard pricing since no real-time latency is required. Combining both captures the maximum available savings.
A developer applies cache_control to a 900-token system prompt when calling Claude Sonnet. On the next identical request, the billing breakdown shows no cache-read tokens, only standard input tokens. What is the MOST likely cause?
Claude models enforce a minimum cacheable prompt length (1024 tokens for Opus and Sonnet, 2048 tokens for Haiku). A 900-token block falls under that threshold, so the API silently processes it as a normal, uncached prompt with no error raised.
An architect maintains a regression suite for an agent that calls a search tool, a calculator tool, and a database tool. After upgrading the underlying model, the suite's final-answer exact-match accuracy stays unchanged, but production users report the agent now requires many more tool calls to reach the same answers, increasing cost and latency. What is the regression suite MOST likely missing?
The suite is blind to this regression because it only checks whether the final answer matches, not how many steps or tool calls were used to get there. A complete agent regression suite needs assertions on trajectory metrics such as tool-call count, tool-call ordering, and step efficiency to catch planning degradations that don't affect final correctness.
A data science team needs to run sentiment classification over 500,000 archived customer transcripts overnight, with results required by 9 AM the next morning. Minimizing API cost is the top priority, and no real-time response is needed. Which approach should the architecture use?
The Message Batches API is designed exactly for this profile: large-volume, non-latency-sensitive, asynchronous workloads, offering a 50% cost discount versus standard pricing with results typically available well within the 24-hour processing window, which comfortably meets the 9 AM deadline.
In an observability platform tracking Claude-based agent traces, which metric MOST directly signals that a model or prompt change has degraded the agent's planning efficiency, even when final-answer accuracy remains unchanged?
Tool-call round trips per completed task directly captures how many planning-and-execution cycles the agent needs to resolve a task. An increase in this count, with accuracy held constant, is the clearest sign of a planning efficiency regression that would otherwise be invisible to accuracy-based metrics alone.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →