Audit LLM evaluation pipelines and deliver prioritized remediation findings.
npx skills add https://github.com/marchatton/agent-skills --skill eval-audit
Please help me install this Agent Skill. Skill: eval-audit Source: https://github.com/marchatton/agent-skills/tree/main/.agents/skills/08-evals/eval-audit Command: npx skills add https://github.com/marchatton/agent-skills --skill eval-audit
This Skill identifies critical flaws in your LLM evaluation pipelines, ensuring your metrics are trustworthy and your AI product is genuinely improving.
Audit the current LLM evaluation pipeline for potential issues and provide a prioritized list of findings.
High-intent search queries and answers about installing and using this skill.
Auditing LLM evaluation pipelines systematically inspects six key areas including error analysis to identify critical problems. It checks eval artifacts like traces, evaluator configs, and metrics dashboards to deliver a prioritized report of findings with concrete next steps for remediation.
Common problems with unvalidated judges include vanity metrics and inadequate human review processes that fail to reflect genuine AI performance. An LLM eval audit inspects judge validation and evaluator design to identify these flaws and provide actionable recommendations for improving reliability.
Yes, you can audit your LLM evaluation setup without existing metrics dashboards. The audit can guide you through setting up a new eval infrastructure from scratch, systematically inspecting pipeline hygiene, labeled data, and human review processes to ensure reliable product improvement tracking.
Checking LLM evaluation metric trustworthiness requires systematically inspecting evaluator design, judge validation, and pipeline hygiene. An audit reviews eval artifacts like traces and labeled data to identify vanity metrics and delivers a prioritized list of findings for remediation.
The best way to find flaws in an inherited LLM evaluation system is running a diagnostic audit across six key areas including error analysis and human review. This inspects existing eval artifacts to produce a prioritized findings report with concrete next steps for improving reliability.
majidraza1228Audit LLM eval pipelines and generate prioritized findings reports.
hamelsmuAudit LLM evaluation pipelines for flaws in error analysis and evaluator design.
ricardorocheApply standardized metrics to LLM evaluation datasets for reproducible assessments.
aws-samplesProvision an AWS CodePipeline and CodeBuild CI/CD stack with Makefile-driven build stages.
