test-builder

An evaluation suite for A/B testing requires generating fixtures, rubrics, and harness configurations to measure LLM quality against baselines. This replaces anecdotal evidence with quantifiable, statistically significant data for reproducible performance validation.

Reproducible LLM benchmarking requires Wilcoxon signed-rank tests, bootstrap confidence intervals, and effect size calculations. Applying these statistical tests ensures benchmarking results are significant and validates quality improvements over baseline responses.

Automate scoring for meta-skill performance validation by generating harness configurations and rubrics. This applies an 8-phase evaluation protocol to produce reproducible evidence, domain sampling, and automated scoring of skill outputs.

Validating a new LLM skill typically requires a 25-fixture suite to prove improvement through A/B testing and pilot validation. This sample size satisfies statistical requirements for reproducible evidence and domain sampling.

Evaluation scaffolding works for comparing prompts by generating fixtures and rubrics against baseline LLM responses. It applies statistical tests to ensure A/B testing results are significant and reproducible.