Audit LLM evaluation pipelines for correctness, coverage, and reliability.
npx skills add https://github.com/kinnerkarmanish/mak --skill eval-audit-kinnerkarmanish
Please help me install this Agent Skill. Skill: eval-audit Source: https://github.com/kinnerkarmanish/mak/tree/main/library/skills/analysis/eval-audit Command: npx skills add https://github.com/kinnerkarmanish/mak --skill eval-audit-kinnerkarmanish
This skill provides a systematic review of LLM evaluation pipelines to catch data quality issues, metric misalignment, judge reliability problems, coverage gaps, and statistical weaknesses before they contaminate decisions.
Run a full audit of your evaluation pipeline to identify data quality, metric validity, judge reliability, coverage gaps, statistics, and integrity issues.
High-intent search queries and answers about installing and using this skill.
An LLM evaluation pipeline audit systematically reviews data quality, metric validity, judge reliability, coverage gaps, statistical rigor, and pipeline integrity. It outputs structured Check/Finding pairs with severity levels to ensure pipeline trust.
LLM evaluation pipeline integrity ensures that datasets, prompts, and results are correctly mapped and free from contamination. Maintaining this integrity prevents statistical weaknesses and metric misalignment from corrupting downstream model decisions.
Checking LLM judge reliability and metric validity involves running multi-diagnostic audits across your evaluation workflow. These audits detect metric misalignment and judge inconsistencies, producing prioritized findings to close coverage gaps.
The best way to identify coverage gaps and statistical weaknesses is to inventory and map the evaluation workflow across datasets, prompts, and results. A full diagnostic audit prioritizes these findings to prevent flawed decisions.
Yes, auditing an LLM evaluation pipeline generates prioritized recommendations based on structured Check/Finding pairs with severity. It also suggests follow-up skills to close identified gaps in the evaluation pipeline.
You should run a full diagnostic audit on LLM evaluation metrics when you need to catch data quality issues, judge reliability problems, or statistical weaknesses before they contaminate your model evaluation decisions.
ResearAIStandardize a reference object for quest evaluation with plan and checklist requirements.
Miosa-osaAudit LLM evaluation pipelines for correctness, coverage, and reliability.
MantisWareAudit LLM evaluation pipelines across six diagnostic areas and rank findings by severity.
nimadorostkarConstruct evaluation sets and implement LLM-as-judge metrics for regression testing.
