skills.rest
imMamdouhaboammar
imMamdouhaboammarCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

agency-model-qa-specialist

Audits machine learning models through replication, calibration testing, and interpretability analysis.

Updated Jul 27, 2026
Search Tags:#model validation#interpretability#shap#psi#model risk#model audit#calibration testing
One-click install
npx skills add https://github.com/imMamdouhaboammar/Mimera --skill agency-model-qa-specialist-immamdouhaboammar
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agency-model-qa-specialist
Source: https://github.com/imMamdouhaboammar/Mimera/tree/main/.agents/skills/specialized-model-qa
Command: npx skills add https://github.com/imMamdouhaboammar/Mimera --skill agency-model-qa-specialist-immamdouhaboammar

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, pandas, scikit-learn, scipy, shap, matplotlib.

What problem does it solve? Validating machine learning models built by others requires independent replication, statistical testing, and interpretability analysis that most teams lack the methodology or rigor to perform consistently. ## Core Features & Use Cases - End-to-End Model Audit: Covers 10 QA domains from documentation review and data reconstruction to calibration testing, fairness analysis, and audit-grade reporting. - Statistical Test Toolkit: Includes ready-to-use implementations of PSI, Gini/KS discrimination metrics, Hosmer-Lemeshow calibration tests, SHAP analysis, and Partial Dependence Plots. - Use Case: A risk team needs to validate a credit scoring model before regulatory review. The skill replicates the training pipeline, tests calibration across subpopulations, runs SHAP feature analysis, and produces a severity-rated findings report. ## Quick Start Audit this XGBoost credit risk model by replicating its training pipeline, testing calibration, and producing a QA report with severity-rated findings.

Frequently Asked Questions about agency-model-qa-specialist

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit a machine learning model independently?▼

Independent model auditing follows four phases: documentation and governance review, data and feature quality checks, model replication with calibration testing, and severity-rated reporting. Every analysis must be reproducible from raw data with versioned scripts.

How to test model calibration with Hosmer-Lemeshow?▼

The Hosmer-Lemeshow test groups predictions into deciles and compares observed versus expected event rates using a chi-square statistic. A p-value below 0.05 indicates significant miscalibration requiring remediation.

What does a PSI value above 0.25 mean for model features?▼

A Population Stability Index of 0.25 or higher indicates significant distribution shift between the baseline and current population, requiring action. Values between 0.10 and 0.25 signal moderate shift warranting investigation.

Does SHAP analysis work with non-tree-based models?▼

SHAP TreeExplainer works natively with XGBoost, LightGBM, and random forests. For other model types, the workflow falls back to KernelExplainer using a sampled background dataset to compute feature contributions.

When should a model be considered unsound in a QA review?▼

A model is unsound when findings are rated High severity, such as failed replication, critical miscalibration, or unstable feature behavior that invalidates predictions. Every finding must quantify impact with evidence rather than opinion.

Related Skills

Explore More Data & Analytics Skills→
writerwriter
6

data-lineage-explanation

Explain data lineage and traceability for financial institution data assets.

Official
Advanced
NUMU-IONUMU-IO
2

speckit-security-review-followup

Convert security review findings into backlog-ready remediation and technical debt plans.

Official
Intermediate
PuckAPIPuckAPI
2

data-pipeline

Automate sports analytics pipelines for data collection, predictions, and drift detection.

Official
Intermediate
kalilurrahmankalilurrahman
1

model-retraining

Automate ML model retraining pipelines with drift triggers, champion-challenger evaluation, and rollback.

Community
Advanced