eval-engineer

To design an evaluation plan for agent workflows, you define the target workflow decisions, identify high-risk failure modes, and translate them into a scenario matrix with objective metrics, pass/fail criteria, and regression thresholds.

Offline evaluation tests prompts against predefined scenario matrices, while live validation monitors agent workflows in production. This Skill helps differentiate these efforts and design appropriate scoring approaches for both validation workflows.

You set regression thresholds by establishing clear pass/fail criteria and objective metrics within a lean evaluation plan. This ensures measurable quality safeguards are in place to catch performance degradation in tool usage.

Yes, you can use human-review-assisted evaluation to assess high-risk failure modes. This Skill designs evaluation plans that explicitly differentiate offline evaluation, human review, and live validation efforts.

The best way to build a scenario matrix is to identify highest-risk failure modes and translate them into measurable scenarios with clear scoring approaches. This creates a rigorous evaluation plan for multi-step agent workflows.