Output Evaluation and Validation
Free Claude Certified Associate - Foundations practice — 6 questions on Output Evaluation and Validation, with explanations. No sign-up. Full 12-question mixed test →
A paralegal uses Claude to draft a legal memo and asks it to cite relevant case law. Claude returns a memo with five case citations, including full case names, years, and court names, all formatted correctly. Which validation step is MOST important before the memo is used?
Hallucinated case citations are a well-documented LLM failure mode. Perfectly formatted citations can still refer to nonexistent or misstated cases, so the only reliable check is independent verification against an authoritative legal database.
During review, an analyst notices that Claude's output states 'According to a 2024 McKinsey study, 73% of enterprises adopted AI governance frameworks,' but no such study or statistic can be found despite the confident, specific tone. What is this error type, and what should the analyst do?
A specific, confidently-stated statistic with a false attribution to a nonexistent study is a classic hallucination. It should be removed or flagged and replaced with a verified source, not reworded or trusted.
A team is building an automated pipeline where Claude extracts invoice fields (vendor, amount, due date) that are parsed by downstream Python code. Which instruction to Claude BEST ensures reliable automated validation of the output?
A fixed JSON schema is machine-parseable and allows automated schema validation (correct field names, types, presence) before downstream processing, catching malformed output before it breaks the pipeline.
Since Claude does not expose a native numeric confidence score with its responses, which approach is the MOST reliable way to decide when an output requires mandatory human review in a production pipeline?
Since LLMs lack reliably calibrated self-confidence, the recommended practice is defining risk-based rules tied to domain impact (e.g., medical, legal, financial) and routing those categories to mandatory human review regardless of surface tone.
A hiring team uses Claude to summarize candidate resumes for a shortlist. Reviewing the outputs, a diversity officer notices summaries describing candidates with traditionally masculine-associated names using words like 'driven' and 'assertive,' while candidates with traditionally feminine-associated names are described as 'organized' and 'supportive' for similar achievements. What is the BEST response?
This reflects proper output bias mitigation: investigate for a systematic pattern across a larger sample, adjust prompting toward objective achievement-based language, and keep human review in the loop for a sensitive, high-impact use case like hiring.
Which statement BEST distinguishes a hallucination from a bias issue in Claude's output?
This correctly separates the two failure modes: hallucination concerns factual fabrication/incorrectness, while bias concerns unfair or unbalanced framing, selection, or representation even when facts are technically accurate.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →