AI Auditing Tools and Techniques
Free Advanced in AI Audit practice — 6 questions on AI Auditing Tools and Techniques, with explanations. No sign-up. Full 12-question mixed test →
An AI audit team is testing a hiring algorithm for adverse impact against a protected class using the four-fifths rule. The selection rate for the protected group is 45%, and for the majority group is 68%. What is the correct determination?
Dividing the protected group's selection rate (45%) by the majority group's rate (68%) yields approximately 0.66, or 66%. Since this falls below the 80% (four-fifths) threshold established by the Uniform Guidelines on Employee Selection Procedures, adverse impact is indicated and warrants further statistical and practical significance testing.
An auditor must evaluate individual prediction-level explanations for a proprietary deep neural network credit risk model, where the vendor contractually restricts access to model internals (true black-box constraint). Which explainability technique is MOST appropriate for producing theoretically consistent, instance-level explanations under this constraint?
SHAP is model-agnostic, works purely from input-output query access (no internal access needed), and is grounded in cooperative game theory, providing consistency and local accuracy guarantees for individual prediction explanations — properties auditors rely on when documenting explainability testing for a specific applicant's decision.
During a third-party AI audit, the vendor supplying a foundation model refuses to disclose training data composition or model weights, citing trade secret protections. Which audit procedure provides the STRONGEST alternative source of assurance over the vendor's AI development and data governance controls?
When direct access to proprietary training data or model internals is contractually or legally restricted, independent third-party attestations (SOC 2 Type II, ISO/IEC 42001) provide auditor-recognized evidence that an external, qualified party has tested the vendor's AI governance and control environment, which is the standard alternative assurance mechanism cited in AAIA guidance for third-party AI risk.
An auditor is testing an organization's model monitoring control, which uses the Population Stability Index (PSI) to detect data drift between the model's training population and current production inputs. Per generally accepted practice, which PSI range indicates a significant distribution shift requiring investigation?
A PSI between 0.10 and 0.25 is widely recognized as indicating moderate-to-significant population shift warranting investigation, while values above 0.25 indicate major shift requiring immediate action; auditors testing a drift-monitoring control should verify the configured alert thresholds align with these accepted benchmarks.
Which statement BEST differentiates independent model validation from an AI audit?
Model validation is a second-line function focused on technical soundness (data quality, methodology, performance) of a specific model, typically performed continuously or at defined intervals, while an AI audit is a third-line, independent assurance activity assessing the adequacy of the entire AI governance, risk, and control ecosystem, including whether validation itself is effective — this distinction is core to AAIA's three-lines-of-defense framing.
During backtesting of a loan approval model, the audit team finds a statistically significant 12 percentage-point drop in approval accuracy for applicants aged 62 and older compared to the overall population. The model's documented fairness policy specifies a maximum acceptable subgroup deviation of 5 percentage points. The model owner argues the finding is immaterial because overall portfolio accuracy remains above 90%. How should the audit team classify this finding in the audit report?
The finding directly violates the organization's own documented control threshold (5 percentage points) with a deviation more than double that limit, and it correlates with a protected age-based subgroup, indicating potential disparate impact — this combination of control breach, statistical significance, and fairness/compliance risk warrants a high-risk finding regardless of aggregate portfolio performance.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →