skills.rest
Muvon
MuvonOFFICIAL
Explore all skills by this author →
Website𝕏@muvonteamGitHub

ai-evals-design

Designs statistically grounded LLM evaluation methodology with paired significance testing and effect sizes.

3|Updated Mar 14, 2026
Search Tags:#llm-as-judge#golden dataset#ragas#drift monitoring#statistical testing#ci regression#llm evals
One-click install
npx skills add https://github.com/Muvon/octomind-tap --skill ai-evals-design
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ai-evals-design
Source: https://github.com/Muvon/octomind-tap/tree/main/skills/ai-evals-design
Command: npx skills add https://github.com/Muvon/octomind-tap --skill ai-evals-design

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Prevents unreliable “looks better” judgments by turning LLM evaluation into a statistically grounded, production-ready methodology you can repeat in CI and use for drift diagnosis.

Core Features & Use Cases

  • Golden dataset design: Construct stratified, version-controlled datasets with power-aware sizing for prompt/model/agent regression detection.
  • Metric + LLM-judge methodology: Select appropriate metrics (including RAG metrics like RAGAS and TruLens RAG Triad) and use G-Eval-style judge scoring to align with human criteria.
  • Significance-driven gating: Apply paired statistical tests (e.g., McNemar’s, paired t-test, Bayesian pairwise, bootstrap) and require effect sizes + confidence intervals before declaring improvements.
  • Agent and RAG coverage: Evaluate agent trajectories, tool use, and RAG faithfulness/retrieval quality to pinpoint failure modes.

Quick Start

Use this skill to design and validate an evaluation suite that gates every prompt change by generating eval verdicts with statistical evidence for your LLM application.

Frequently Asked Questions about ai-evals-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design LLM evals to detect prompt regression in CI?▼

To design LLM evals for CI regression, construct a stratified, version-controlled golden dataset with power-aware sizing, then gate prompt and model changes using statistically valid paired significance testing with effect sizes and confidence intervals.

What is LLM-as-judge evaluation and how do I mitigate preference leakage?▼

LLM-as-judge evaluation uses G-Eval-style scoring to align with human criteria, requiring preference leakage mitigation techniques to ensure the judge model does not favor specific outputs during paired comparisons.

How do I evaluate RAG faithfulness and retrieval quality?▼

Evaluate RAG faithfulness and retrieval quality by selecting appropriate metrics like RAGAS and TruLens RAG Triad to pinpoint failure modes in generation and retrieval within your LLM evaluation methodology.

Which statistical tests should I use for paired LLM evaluation comparisons?▼

Paired LLM evaluation comparisons require statistical tests like McNemar’s, paired t-test, Bayesian pairwise, or bootstrap, combined with effect sizes and confidence intervals to validate improvements.

What is the best way to evaluate agent trajectories and tool use?▼

Evaluating agent trajectories and tool use requires production-grade methodology that assesses multi-step execution paths and failure modes, replacing subjective quality claims with evidence-driven metrics.

How much data do I need for a golden dataset to detect model drift?▼

Golden datasets for drift monitoring require power-aware sizing to ensure statistically significant regression detection, using stratified and version-controlled samples to accurately diagnose model performance changes.

Related Skills

Explore More Software Engineering Skills→
Sir-chawakornSir-chawakorn
9

llm-eval-harness

Builds an LLM evaluation harness using golden datasets and code-based scorers.

Community
Advanced
rocklambrosrocklambros
2

checking-test-assumptions

Check statistical-test assumptions and issue pass-fail verdicts with evidence.

Community
Advanced
MuvonMuvon
3

ai-rag-patterns

Design and troubleshoot production RAG pipelines with chunking, hybrid retrieval, reranking, and evaluation.

Official
Advanced
aisa-groupaisa-group
90

llm-evaluation

Compute automated metrics and statistical tests for LLM evaluation.

Official
Advanced