evaluation

An evaluation suite standardizes how you test AI outputs by combining code-based validation (deterministic checks) and LLM-as-judge assessment (quality scoring). Start with spec.md to document what to test, then rubric.md to define objective criteria, ensuring consistent, measurable results across iterations.

Code-based validation runs deterministic checks—pass/fail rules you hard-code. LLM-as-judge uses an AI model to assess subjective qualities like clarity or correctness against your rubric. Combined, they catch both objective failures and quality gaps that code alone misses.

Objective rubrics reduce subjectivity by defining concrete, verifiable criteria instead of vague guidelines. Specify measurable thresholds, examples of pass/fail outputs, and what the LLM should score. This consistency improves reliability and makes results reproducible across evaluation runs.

Create a new suite when building a feature you haven't tested before; spec and rubric define what quality means. Update an existing suite when requirements change, validation gaps appear, or you need to add LLM-judge checks. Both apply the same standardized structure.

Yes. A standardized evaluation suite with consistent metrics (code-based scores and rubric-based judgments) lets you compare model performance across versions. Spec.md and rubric.md remain stable while test data or model versions change, enabling reliable performance tracking.

Both serve different purposes: spec.md describes what features and edge cases to test; rubric.md defines scoring criteria for LLM evaluation. Using both ensures comprehensive coverage—spec guides test selection, rubric ensures consistent judgment quality.