evaluation

Define weighted dimensions such as factual accuracy, completeness, and tool efficiency, then score each output per dimension and compute a weighted overall score. The AgentEvaluator class applies a default five-dimension rubric and returns per-dimension scores with a pass/fail verdict at the 0.7 threshold.

Start with 20-30 cases during early development and scale to at least 50 for reliable signal. Stratify cases across simple, medium, complex, and very complex levels, sample from real usage patterns, and report scores per stratum so easy cases do not mask failures.

LLM-as-judge uses a language model to score agent outputs against a rubric with explicit level descriptions, enabling consistent evaluation across large test sets. Use a different model family as the judge than the agent being evaluated to avoid self-enhancement bias, and supplement with human review.

One aggregate number averages away dimension-specific problems, such as high accuracy with poor tool efficiency. Multi-dimensional rubrics expose which dimension needs improvement, and evaluations should fail if any single dimension falls below its minimum threshold.

Sample a fraction of live interactions, evaluate each sample, and track pass rate and average score over time. The ProductionMonitor class triggers a warning below a 0.85 pass rate and a critical alert below 0.70, helping detect gradual quality drift.

Automated evaluation misses hallucinations, subtle biases, and edge cases that human reviewers catch. It is also vulnerable to test set contamination and metric gaming, so eval sets should stay versioned and separate from training data, with regular human cross-validation.