Audit AI models with evaluation, calibration, and SHAP analysis.
npx skills add https://github.com/coreymaypray/sloth-skill-tree --skill model-qa
Please help me install this Agent Skill. Skill: model-qa Source: https://github.com/coreymaypray/sloth-skill-tree/tree/main/plugins/maycrest-automate/skills/model-qa Command: npx skills add https://github.com/coreymaypray/sloth-skill-tree --skill model-qa
Audits ML and AI models throughout their lifecycle, focusing on LLM evaluation, prompt quality, and governance to ensure safe, reliable outputs.
Run a structured QA audit on a target model using predefined evaluation prompts and generate a reproducible report.
High-intent search queries and answers about installing and using this skill.
Automate LLM evaluation by running structured QA audits with standardized prompts, scoring, and reporting to ensure reliable prompt quality. This framework supports adversarial testing, fairness audits, and reproducible methodology documentation.
An end-to-end model QA audit includes LLM evaluation, prompt quality checks, fairness audits, and interpretability analyses. It leverages SHAP analysis, PSI, calibration tests, and generates audit-grade reports for stakeholder review.
Yes, you can generate audit-grade reports for governance and compliance by documenting methodology replication and maintaining a model inventory. This ensures stakeholder review and reproducible pipelines across development and production environments.
Run fairness checks and interpretability analyses by applying standardized evaluation prompts and calibration tests to AI models. This process includes SHAP analysis and PSI to evaluate model behavior and ensure safe, reliable outputs.
Yes, the framework supports reproducible pipelines by applying standardized prompts and methodology documentation throughout the model lifecycle. This ensures consistent LLM testing, calibration checks, and audit-grade reporting across environments.
The best way to perform calibration tests on LLM outputs is using a structured QA evaluation framework with predefined prompts. This approach ensures reproducible methodology, fairness audits, and standardized scoring for accurate model assessment.
selcukyucelGenerate structured evaluation datasets with binary rubric scoring for AI outputs.
skwuwuVerify GVM audit log integrity and export compliance reports.
rajyeole6Audit machine learning models by replicating training pipelines and computing PSI, Gini, KS, and SHAP metrics.
HermeticOrmusGenerate classification metrics, calibration analysis, and fairness evaluations for machine learning models.
