evaluator-design-new

Name: evaluator-design-new
Availability: InStock
Author: agenticgogol

Route failure modes to code checks or LLM judges and write pytest evaluators.

1|9|Updated Jul 11, 2026

Search Tags:#pytest #routing #evaluation #failure modes #llm judge #agent traces #code eval

Ready to run

npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill evaluator-design-new-agenticgogol

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: evaluator-design-new
Source: https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul/tree/main/.claude/skills/evaluator-design-new
Command: npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill evaluator-design-new-agenticgogol

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? After axial coding produces a taxonomy of agent failure modes, teams must decide which failures can be caught by deterministic code checks and which require an LLM judge, then actually implement the code checks. This Skill makes that routing decision systematically and writes the pytest-style evaluators. ## Core Features & Use Cases - Routing Decisions: Classifies each failure mode in evals/taxonomy.yaml as code or judge using documented criteria, recording a one-line rationale per decision in evals/evaluators/routing.yaml. - Code Eval Generation: Writes pytest-style check functions that take a Trace and return a boolean, validated against known-failing and known-passing traces from evals/traces.jsonl. - Mix Enforcement: Targets roughly 2-3 code evals per 1-2 judges and flags backlogs that skew too heavily toward judges. - Use Case: After running axial coding on agent traces, use this Skill to decide that "skipped eligibility check" is a tool-order code check while "tone mismatch" needs a judge, then generate the code eval for the former. ## Quick Start Route the failure modes in evals/taxonomy.yaml to code checks or judges and write the code evaluators.

Frequently Asked Questions about evaluator-design-new

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I decide between a code check and an LLM judge for an eval?▼

Default to code when the criterion is structural: regex or string match, JSON schema validation, set membership, tool-call order, or numeric thresholds. Route to a judge only when the criterion is genuinely subjective, such as tone, comprehensibility, or plan soundness.

How do I write a code eval for an agent failure mode?▼

Write a pytest-style function that takes a Trace and returns True if the trace passes. Base the check on the structural signal named in the routing rationale, then validate it against the failure mode's member traces (should fail) and non-member traces (should pass).

What is a good ratio of code evals to LLM judges?▼

Target roughly 2-3 code evals per 1-2 judges across a backlog. A mostly-judge backlog is a warning sign to re-read each failure mode definition for a missed structural signal before accepting the judge route.

When should I not use an LLM judge for evaluation?▼

Avoid judges when any mechanical check exists, since judges cost money per run, can drift, and require calibration before they are trustworthy. Judges are justified only for subjective readings no regex, schema, or threshold check can replace.

What inputs does the evaluator routing step require?▼

It requires evals/taxonomy.yaml and evals/backlog.md produced by the axial coding phase. If either file is missing, the process stops and reports the gap rather than routing from incomplete information.