eval-engineer

Design evaluation plans for prompts, tools, and multi-step agent workflows.

22|2|Updated Mar 24, 2026

Search Tags:#metrics #prompts #evaluate #agent-workflows #evaluation-design #scenario-matrix

Ready to run

npx skills add https://github.com/jshsakura/awesome-opencode-skills --skill eval-engineer

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-engineer
Source: https://github.com/jshsakura/awesome-opencode-skills/tree/main/skills/eval-engineer
Command: npx skills add https://github.com/jshsakura/awesome-opencode-skills --skill eval-engineer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps teams design rigorous evaluation plans for prompts, tool usage, and multi-step agent workflows to ensure measurable quality and reliability.

Core Features & Use Cases

Define the target workflow and its decisions to support evaluation.
Identify highest-risk failure modes and translate them into measurable scenarios.
Develop a lean evaluation plan with clear pass/fail criteria and regression safeguards.
Differentiate offline evaluation, human review, and live validation efforts.

Quick Start

Draft and execute a lean evaluation plan for a given prompt workflow and verify it against defined success metrics.

Frequently Asked Questions about eval-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design an evaluation plan for multi-step agent workflows?▼

To design an evaluation plan for agent workflows, you define the target workflow decisions, identify high-risk failure modes, and translate them into a scenario matrix with objective metrics, pass/fail criteria, and regression thresholds.

What is the difference between offline evaluation and live validation for prompts?▼

Offline evaluation tests prompts against predefined scenario matrices, while live validation monitors agent workflows in production. This Skill helps differentiate these efforts and design appropriate scoring approaches for both validation workflows.

How do I set regression thresholds for prompt and tool usage evaluation?▼

You set regression thresholds by establishing clear pass/fail criteria and objective metrics within a lean evaluation plan. This ensures measurable quality safeguards are in place to catch performance degradation in tool usage.

Can I use human review to evaluate high-risk failure modes in agent workflows?▼

Yes, you can use human-review-assisted evaluation to assess high-risk failure modes. This Skill designs evaluation plans that explicitly differentiate offline evaluation, human review, and live validation efforts.

What's the best way to build a scenario matrix for evaluating agent workflows?▼

The best way to build a scenario matrix is to identify highest-risk failure modes and translate them into measurable scenarios with clear scoring approaches. This creates a rigorous evaluation plan for multi-step agent workflows.