eval-audit

Name: eval-audit
Availability: InStock
Author: Miosa

Audit LLM evaluation pipelines for correctness, coverage, and reliability.

226|55|Updated Mar 19, 2026

Search Tags:#audit #llm #metrics #evaluation #reliability #bias #data-quality

Ready to run

npx skills add https://github.com/Miosa-osa/canopy --skill eval-audit-miosa-osa

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-audit
Source: https://github.com/Miosa-osa/canopy/tree/main/library/skills/analysis/eval-audit
Command: npx skills add https://github.com/Miosa-osa/canopy --skill eval-audit-miosa-osa

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Systematically auditing LLM evaluation pipelines to uncover data quality issues, metric misalignment, judge reliability gaps, coverage blind spots, and pipeline integrity risks before decisions are made on model performance.

Core Features & Use Cases

Six diagnostic areas (data quality, metric validity, judge reliability, coverage, statistics, and integrity) with structured Check/Finding outputs.
Severity-ranked findings and prioritized recommendations to guide remediation and follow-up skills.
Works across end-to-end evaluation pipelines and supports generating actionable takeaways for model development teams.

Quick Start

Run /eval-audit --pipeline evals/ to perform a full audit of an evaluation pipeline.

Frequently Asked Questions about eval-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I audit an LLM evaluation pipeline for correctness and reliability?▼

You can audit an LLM evaluation pipeline by running a structured diagnostic across data quality, metric validity, judge reliability, coverage, statistics, and integrity to identify gaps and prioritize fixes.

What is LLM evaluation pipeline coverage and why does it matter?▼

LLM evaluation pipeline coverage refers to the breadth of diagnostic checks applied across datasets, metrics, and judges. It matters because coverage blind spots can mask model performance issues before decisions are made.

How do I check judge reliability in my LLM evaluation metrics?▼

To check judge reliability in LLM evaluation metrics, run a structured audit that evaluates judge consistency and alignment, producing severity-ranked findings to guide remediation efforts.

Can I assess data quality and bias issues in LLM evaluation datasets?▼

Yes, you can assess data quality and bias issues in LLM evaluation datasets by applying a structured audit that diagnoses data quality gaps and outputs actionable findings for model development teams.

What's the best way to identify metric misalignment in an LLM eval workflow?▼

The best way to identify metric misalignment in an LLM eval workflow is to run an end-to-end audit that checks metric validity, producing prioritized recommendations and follow-up skills to correct the gaps.

When should I audit my LLM evaluation pipeline integrity?▼

You should audit your LLM evaluation pipeline integrity before making decisions on model performance, as auditing uncovers pipeline integrity risks, coverage blind spots, and data quality issues early.