ai-evals-design

What problem does it solve?

Prevents unreliable “looks better” judgments by turning LLM evaluation into a statistically grounded, production-ready methodology you can repeat in CI and use for drift diagnosis.

Core Features & Use Cases

• Golden dataset design: Construct stratified, version-controlled datasets with power-aware sizing for prompt/model/agent regression detection.
• Metric + LLM-judge methodology: Select appropriate metrics (including RAG metrics like RAGAS and TruLens RAG Triad) and use G-Eval-style judge scoring to align with human criteria.
• Significance-driven gating: Apply paired statistical tests (e.g., McNemar’s, paired t-test, Bayesian pairwise, bootstrap) and require effect sizes + confidence intervals before declaring improvements.
• Agent and RAG coverage: Evaluate agent trajectories, tool use, and RAG faithfulness/retrieval quality to pinpoint failure modes.

Quick Start

Use this skill to design and validate an evaluation suite that gates every prompt change by generating eval verdicts with statistical evidence for your LLM application.