hamelsmuCommunity·7 Agent Skills Included

evals-skills

Build trustworthy LLM evaluations with proven audit and testing workflows

Audits LLM evaluation pipelines and surfaces problems like unvalidated judges, vanity metrics, and missing error analysis. Guides failure categorization, synthetic test data generation, LLM-as-Judge prompt design, and judge calibration against human labels. Eliminates guesswork in AI quality testing so teams ship reliable models faster.

1,641|164

↗

npx skills add hamelsmu/evals-skills --all -g -y

All Skills in This Repository (7)

Pure Emerald Level Indicators

📦 In Repo

hamelsmu

error-analysis

Analyze LLM pipeline traces to categorize and prioritize failure modes.

build-review-interface

Generate a browser-based annotation interface for reviewing LLM traces with Pass/Fail feedback.

write-judge-prompt

Design LLM-as-Judge evaluators for subjective criteria with binary pass/fail outcomes.

eval-audit

Audit LLM evaluation pipelines for flaws in error analysis and evaluator design.

evaluate-rag

Evaluate RAG pipelines by measuring retrieval metrics and generation quality.

validate-evaluator

Calibrate LLM judges against human labels using TPR/TNR measurement and bias correction.

generate-synthetic-data

Generate diverse synthetic test inputs for LLM pipeline evaluation.

Community

Intermediate

Frequently Asked Questions

FAQPage Schema

How to install evals-skills?▼

Run `npx skills add hamelsmu/evals-skills --all -g -y` in your terminal to install all evaluation skills globally.

How to audit an LLM eval pipeline?▼

Use the eval-audit skill, which inspects your traces, judge prompts, and labeled data, then reports prioritized problems with concrete fixes.

How to build a reliable LLM-as-Judge evaluator?▼

The write-judge-prompt skill designs binary pass/fail judges with few-shot examples, and validate-evaluator calibrates them against human labels using TPR/TNR.

Can I evaluate a RAG pipeline with these skills?▼

Yes. The evaluate-rag skill measures retrieval quality with Recall@k and MRR, then separately checks generation faithfulness and relevance.

Do these skills work with Claude Code and other AI agents?▼

Yes. All skills follow the standard SKILL.md format and run in Claude Code, Cursor, and other compatible AI coding agents.