evaluation

You evaluate agent performance by scoring outputs across multiple dimensions like factual accuracy, completeness, and tool efficiency separately, aggregating weighted scores with pass/fail thresholds to detect regressions instead of guessing.

LLM-as-judge evaluation uses structured rubric prompts to assess agent outputs at scale, providing structured judgments on quality dimensions while supplementing edge cases and subtle failures with human sampling review.

You test non-deterministic agents by evaluating results against outcome criteria rather than fixed execution paths, allowing agents to vary their routes while still being judged fairly on whether they meet quality thresholds.

Yes, you can track pass rates over time using production sampling-based metric reporting and block or alert on quality drops by configuring pass/fail thresholds as quality gates in your deployment pipeline.

You should score factual accuracy, completeness, citation accuracy, source quality, and tool efficiency separately rather than relying on a single number, aggregating these weighted dimensions to measure overall agent quality.

Yes, you need a test set with outcome-based scoring criteria to evaluate whether a new agent version meets your acceptance thresholds for quality and efficiency before deploying it to production.