Manage AI Model Development and Evaluation
Free PMI Certified Professional in Managing AI practice — 6 questions on Manage AI Model Development and Evaluation, with explanations. No sign-up. Full 12-question mixed test →
A project team is preparing to move from data preparation into model development. During final data quality review, the team discovers that 15% of records in the training dataset have missing values in a feature identified as highly predictive during business understanding, and the missingness is not random but correlates strongly with a specific customer segment. What is the BEST course of action regarding the go/no-go gate decision?
Non-random (MNAR) missingness correlated with a segment risks introducing systematic bias into the model. The gate discipline requires the root cause be investigated and remediated, not merely accepted or deferred, before advancing past the data quality go/no-go checkpoint.
During final evaluation for the operationalization go/no-go gate, a model achieves 94% accuracy on the held-out test set, meeting the primary success criterion set during business understanding. However, on a stress-test dataset representing seasonal variation and rare edge cases, performance drops to 61%. The sponsor is eager to deploy immediately given the headline accuracy figure. What should the team recommend?
The operationalization gate explicitly requires evaluating performance against success criteria AND robustness/generalization AND deployment readiness with documented approval. A large performance drop on edge cases and seasonal variation fails the robustness criterion, so the gate should not pass regardless of the headline test-set number.
A logistics company wants a system to determine optimal delivery route sequencing. There is no historical dataset of 'correct' routes, and the system only receives feedback (total fuel cost and delivery time) after an entire day's route sequence is completed in a simulated environment. Which category of technique is MOST appropriate for this use case?
Sequential decision-making with delayed, cumulative feedback and no ground-truth labels is the defining characteristic of a reinforcement learning problem, where an agent learns a policy via trial-and-error interaction with an environment.
A data science team ran 40 experiments over three weeks, adjusting hyperparameters and feature sets. The AI project manager later asks why experiment version 27 outperformed version 34 despite similarly reported metrics, but no one can reconstruct what data snapshot, code version, or hyperparameters were used for version 34. Which practice should the project manager have enforced from the start to prevent this?
Experiment tracking and data/model versioning that captures data snapshots, code versions, hyperparameters, and metrics for every run is the core practice for managing training reproducibility and enabling apples-to-apples comparison across experiments.
During a peer review of model development artifacts, a senior data scientist discovers that the team performed hyperparameter tuning and feature selection using the same held-out test set that was later used to report the model's final performance metrics. What is the MOST significant risk this introduces, and what should the reviewer recommend?
Using the test set for tuning decisions is a form of data leakage that invalidates the final performance metric as an unbiased estimate of real-world performance. Correct remediation is a proper train/validation/test split, where the test set is untouched until final evaluation, followed by re-evaluation.
A healthcare AI project is developing a model to support clinical treatment recommendations, and regulatory requirements mandate that clinicians be able to understand the reasoning behind each recommendation. A deep neural network ensemble achieves 96% accuracy, while a rule-based decision tree achieves 89% accuracy but produces human-readable decision paths. According to CPMAI principles on technique selection trade-offs, what should guide the final model selection?
CPMAI's complexity-performance-interpretability trade-off principle requires evaluating technique selection against the specific use case and regulatory context, not accuracy alone. When explainability is a hard regulatory requirement and the accuracy gap is modest, the more interpretable model is generally the appropriate choice.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$109.99 $34.99 with code FREETEST33 — valid through September 14.
Get my $34.99 deal →