Audit LLM evaluation pipelines for correctness, coverage, and reliability.
npx skills add https://github.com/Miosa-osa/canopy --skill eval-audit-miosa-osa
Please help me install this Agent Skill. Skill: eval-audit Source: https://github.com/Miosa-osa/canopy/tree/main/library/skills/analysis/eval-audit Command: npx skills add https://github.com/Miosa-osa/canopy --skill eval-audit-miosa-osa
Systematically auditing LLM evaluation pipelines to uncover data quality issues, metric misalignment, judge reliability gaps, coverage blind spots, and pipeline integrity risks before decisions are made on model performance.
Run /eval-audit --pipeline evals/ to perform a full audit of an evaluation pipeline.
High-intent search queries and answers about installing and using this skill.
You can audit an LLM evaluation pipeline by running a structured diagnostic across data quality, metric validity, judge reliability, coverage, statistics, and integrity to identify gaps and prioritize fixes.
LLM evaluation pipeline coverage refers to the breadth of diagnostic checks applied across datasets, metrics, and judges. It matters because coverage blind spots can mask model performance issues before decisions are made.
To check judge reliability in LLM evaluation metrics, run a structured audit that evaluates judge consistency and alignment, producing severity-ranked findings to guide remediation efforts.
Yes, you can assess data quality and bias issues in LLM evaluation datasets by applying a structured audit that diagnoses data quality gaps and outputs actionable findings for model development teams.
The best way to identify metric misalignment in an LLM eval workflow is to run an end-to-end audit that checks metric validity, producing prioritized recommendations and follow-up skills to correct the gaps.
You should audit your LLM evaluation pipeline integrity before making decisions on model performance, as auditing uncovers pipeline integrity risks, coverage blind spots, and data quality issues early.
trueraOrchestrates end-to-end TruLens evaluation workflows for LLM apps.
kinnerkarmanishAudit LLM evaluation pipelines for correctness, coverage, and reliability.
MantisWareAudit LLM evaluation pipelines across six diagnostic areas and rank findings by severity.
RHEcosystemAppEngOrchestrate containerization and deployment to OpenShift or RHEL systems.
