evaluation

Score outputs against a multi-dimensional rubric covering factual accuracy, completeness, citation accuracy, source quality, and tool efficiency. Compute a weighted overall score and apply a pass threshold, typically 0.7 for general use and 0.9 for high-stakes applications.

Start with 20-30 cases during early development and scale to at least 50 for reliable signal. Stratify cases across simple, medium, complex, and very complex levels, sample from real usage patterns, and report scores per stratum so easy cases do not inflate results.

Run deterministic validation first whenever the artifact has machine-checkable structure, such as schema validity or required files. Apply LLM judgment only after deterministic checks and rubrics are stable, and use a different model family as judge to avoid self-enhancement bias.

Sample a fraction of production interactions, evaluate each against the rubric, and track pass rate over time. Set alerts when pass rate falls below 0.85 (warning) or 0.70 (critical), and review dashboards for gradual degradation trends.

A single number hides dimension-specific failures, such as high accuracy paired with poor tool efficiency. Always report per-dimension scores alongside the aggregate and fail the evaluation if any single dimension drops below its minimum threshold.

Common failures include overfitting tests to specific execution paths, test set contamination, metric gaming, and eval sets under 50 cases. Evaluate outcomes rather than steps, keep eval data separate from prompts, and cross-validate automated metrics with human review.