agency-model-qa-specialist

To audit machine learning models for data drift and calibration, run an end-to-end QA process that computes Population Stability Index (PSI) for drift, Hosmer-Lemeshow tests for calibration, and discrimination metrics to validate model reliability across development and production scenarios.

Model interpretability analysis using SHAP and PDP involves generating global and local SHAP analyses and Partial Dependence Plots to detect feature interactions and explain predictions. This independent audit uncovers interpretability blind spots that could cause production failures.

Yes, you can run a model QA audit on NLP and computer vision models. The audit process applies to classification, regression, ranking, recommendation, forecasting, NLP, and computer vision models across development, out-of-time, and production evaluation scenarios.

To check model fairness across protected groups, perform an interpretability and fairness audit that generates SHAP analyses and runs fairness checks across protected groups. This identifies governance issues and produces severity-rated findings with quantified impact and remediation recommendations.

For a reproducible model QA audit, you need to provide model artifacts, training and out-of-time (OOT) data snapshots, and methodology documents. The audit then produces reproducible scripts, delta reports comparing parameter and score deltas, and a governance-ready audit report.

You should benchmark challenger models against a production model during a model replication and calibration audit to reproduce training, compare parameter and score deltas, and run discrimination metrics. This validates performance regressions and undocumented assumptions causing production failures.