AI Development Lifecycle
Free IAPP AI Governance Professional (AIGP) practice — 6 questions on AI Development Lifecycle, with explanations. No sign-up. Full 12-question mixed test →
A healthcare AI team is developing a diagnostic model and needs to ensure compliance with medical device regulations while maintaining model performance tracking. The team has chosen to implement continuous monitoring post-deployment. During the model validation phase, they discover that their holdout test set of 500 patients shows 94% accuracy, but when tested on data from a different hospital system, accuracy drops to 78%. Which action should be prioritized to address this issue before deployment?
The 16-point accuracy drop indicates dataset shift and lack of generalization across different data distributions. Federated learning specifically addresses this by training across multiple hospital systems (different data distributions) while maintaining privacy compliance required for healthcare data. This directly solves the root cause: the model was trained on data from only one distribution.
An AI product team is implementing a feature store for their recommendation system that serves 10 million users. They need to decide on the feature computation strategy. Real-time user behavior features (last 5 clicks) must be available within 100ms, while user demographic features update monthly. The system currently recomputes all features on every prediction request, causing 800ms average latency. Which architectural approach BEST optimizes this feature pipeline?
This correctly separates feature computation by freshness requirements: demographics (monthly updates) can be precomputed and cached, while behavioral features (requiring real-time data) use stream processing. This hybrid approach meets the 100ms requirement for real-time features while avoiding unnecessary recomputation of static features, directly addressing the 800ms latency problem.
A financial services company is conducting a model risk assessment for their credit scoring AI system before deployment. The model uses 200 features including income, credit history, and behavioral data. During validation, the team discovers that the model's Gini coefficient is 0.68 on the training set but 0.71 on the test set. The data science team argues this indicates a robust model ready for deployment. What is the PRIMARY concern that should be raised during the risk assessment?
Better performance on the test set than training set is a major red flag indicating data leakage, target leakage, or improper train/test split. In proper ML development, test performance should be equal to or slightly worse than training performance due to the model not having seen test data. This suggests information from the target variable or future data leaked into features, creating an invalid model that won't perform as expected in production.
An autonomous vehicle company is implementing a CI/CD pipeline for their perception model updates. The pipeline must ensure safety before deploying to vehicles. Current process: unit tests (5 min) → integration tests (30 min) → shadow mode deployment (7 days) → full deployment. The team wants to reduce the 7-day shadow mode period. Which validation gate would be MOST appropriate to add before reducing shadow mode duration while maintaining safety standards?
Simulation-based testing with comprehensive scenario coverage can validate model behavior across thousands of edge cases in hours rather than days, while maintaining safety standards. This addresses the need to reduce shadow mode duration by front-loading validation. Autonomous vehicle safety requires testing rare scenarios that may not occur during 7-day shadow mode, making simulation critical for safety-critical systems.
A legal tech company is building a contract analysis AI that must be auditable for regulatory compliance. They are deciding on their experiment tracking approach. The system processes 50,000 contracts monthly, and models are retrained quarterly. Regulators require the ability to reproduce any model version's predictions on specific contracts from up to 5 years ago. Which experiment tracking strategy BEST meets these requirements?
Full reproducibility for 5 years requires tracking all components: code (Git commits), hyperparameters, model artifacts (versioned storage), AND dataset snapshots with cryptographic hashes. The hash linking ensures exact dataset reconstruction. This enables reproducing any historical model's predictions exactly, meeting regulatory requirements for auditability. Dataset snapshots are critical because contracts and training data may change over time.
A fraud detection system uses ensemble learning with 5 models updated on different schedules: Model A (daily), Model B (weekly), Model C (monthly), Models D and E (quarterly). The ensemble voting mechanism equally weights all models. After deployment, the system's precision dropped from 0.85 to 0.72 over 6 weeks, while individual model metrics remain stable in offline evaluation. Which diagnostic approach would MOST effectively identify the root cause?
The key insight: individual models show stable offline metrics but ensemble performance degrades in production. This pattern strongly suggests data drift—production data has shifted from training distributions. Since models update on different schedules (daily to quarterly), they're trained on different time periods of data. Data drift would affect all models but go undetected in offline eval if evaluation data is also stale. Checking feature distribution drift directly diagnoses this mismatch.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$109.99 $34.99 with code FREETEST33 — valid through August 22.
Get my $34.99 deal →