AI Operations
Free Advanced in AI Audit practice — 6 questions on AI Operations, with explanations. No sign-up. Full 12-question mixed test →
An auditor reviewing an AI model deployment pipeline finds that a new fraud-detection model was deployed directly to 100% of production traffic without a shadow or canary period, although the model card documented an offline test accuracy of 94%. Which control gap should the auditor cite as the MOST significant risk?
A staged rollout (canary or shadow deployment) is the primary control that validates a model's behavior against live production data and traffic patterns before full exposure, catching issues offline testing cannot reveal (e.g., data skew, latency, edge cases). Skipping this step is the most significant operational risk here.
Per AAIA guidance on reproducibility controls, which set of metadata elements is MINIMALLY required in a model registry entry to support an audit trail for a production ML model?
Reproducibility and audit trail requirements demand traceability from the deployed artifact back to the exact data, code, and configuration used to produce it: dataset version/hash, hyperparameters, code commit reference, and evaluation metrics collectively allow an auditor to reconstruct and validate model lineage.
During an audit of a credit-scoring model's monitoring dashboard, the auditor observes the Population Stability Index (PSI) for a key input feature has risen from 0.05 to 0.28 over the past quarter, with no investigation ticket or retraining triggered. Which conclusion is MOST appropriate?
PSI is a widely used data-drift metric where values below ~0.1 indicate no significant shift, 0.1-0.25 indicate moderate shift warranting attention, and above ~0.25 indicates significant population shift requiring investigation. The absence of any triggered review at this threshold is a clear monitoring-control gap.
An organization trains models using batch feature engineering scripts but computes equivalent features at inference time using separate, independently maintained real-time code, resulting in unexplained model performance degradation in production. Which control would BEST address this audit finding?
A centralized feature store enforces a single, versioned source of feature logic used identically for both training and serving, directly eliminating training-serving skew, which is the root cause of the degradation described.
An auditor is evaluating operational controls for a high-risk generative AI system used in medical triage recommendations. The organization performed a single red-team exercise prior to initial launch 18 months ago and has not repeated it since, despite two subsequent prompt-template and model version updates. What is the MOST appropriate audit finding?
For high-risk generative AI systems, adversarial/red-team testing is an ongoing control that must be repeated after material changes (model version updates, prompt template changes) since these can introduce new failure modes or vulnerabilities; a single pre-launch exercise is insufficient.
Which statement correctly distinguishes data drift from concept drift in the context of MLOps monitoring controls an AI auditor should expect to see implemented?
Data drift concerns shifts in the statistical distribution of input features over time (e.g., via PSI or KL divergence), whereas concept drift concerns a change in the underlying relationship between inputs and the target label (e.g., detected via degrading model accuracy/label-based metrics). Because they arise from different causes, robust MLOps monitoring implements distinct detection mechanisms for each.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →