skills.rest
novuhq
novuhqOFFICIAL
Explore all skills by this author →
Website𝕏@novuhqGitHub

triage-agent-eval-failures

Diagnose failing agent-evals scenarios and classify failures as real regressions, test bugs, or flaky runs.

39.7k|4.4k|Updated Aug 26, 2021
Search Tags:#Vitest#triage#LLM evaluation#evals#grader#novu#Flaky tests
One-click install
npx skills add https://github.com/novuhq/novu --skill triage-agent-eval-failures
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: triage-agent-eval-failures
Source: https://github.com/novuhq/novu/tree/main/.cursor/skills/triage-agent-eval-failures
Command: npx skills add https://github.com/novuhq/novu --skill triage-agent-eval-failures

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

When an @novu/agent-evals scenario goes red, it is unclear whether the playbook under test regressed, the test scaffolding (grader, tape, scenario, judge) is broken, or the failure is just model non-determinism. This Skill provides a structured triage workflow to reach the correct verdict without fixing the wrong layer.

Core Features & Use Cases

  • Flakiness screening: Re-run the failing scenario 3-5 times with vitest before changing anything, separating deterministic failures from flaky ones.
  • Grader classification: Identify whether the failing grader is deterministic (catalog, contains, matches) or an LLM judge, and map it to the RunResult fields it inspects.
  • Verdict-driven fixes: Apply one bounded fix to either the playbook (agent-onboarding.md) or the test layer (catalog.ts, scenario.ts, judge prompts), never both, then verify across re-runs.
  • Use Case: A CI run shows the dashboard-prompt-login scenario failing on the readAuthUrlFile grader. Use this Skill to inspect the RunResult evidence, discover the agent behaved correctly but the grader's path check is too narrow, and fix catalog.ts instead of the playbook.

Quick Start

Ask the agent to triage why the agent-evals scenario dashboard-prompt-login is failing and decide whether to fix the playbook or the test.

Frequently Asked Questions about triage-agent-eval-failures

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I triage a failing agent-evals scenario in Novu?▼

Re-run the single failing scenario 3-5 times with vitest using the evals config to rule out flakiness first. Then identify which grader failed, inspect the RunResult evidence, and classify the failure as a real regression, test bug, or flaky judge before changing anything.

How do I decide whether to fix the playbook or the test?▼

If the RunResult shows the agent genuinely did the wrong thing, fix the playbook agent-onboarding.md. If the agent behaved correctly but the grader, tape, scenario, or judge rejected valid behavior, fix the test layer. Never change both to chase a green run.

Why does an agent eval fail intermittently between runs?▼

Intermittent failures usually come from non-deterministic LLM judge graders or over-strict regex checks, since scenarios run a live model concurrently. Sharpen the judge prompt with explicit pass/fail examples or accept variance with pass@k instead of editing the playbook.

What does a judge UNKNOWN verdict mean in agent-evals?▼

A judge returning UNKNOWN is converted to a skip and scores as 1, so it never causes a failure. An UNKNOWN result is not evidence of a real regression and should not trigger a playbook change.

What is the pass threshold for agent-evals scenarios?▼

A scenario passes only when every active grader averages at least 0.8, defined by JUDGE_THRESHOLD. Deterministic graders inspect structured RunResult fields, while judge graders use a second LLM pass whose verdicts can vary between runs.

Related Skills

Explore More Software Engineering Skills→
davduncdavdunc

Evals

Run regression and capability evaluations for multi-turn AI agent workflows.

Community
Advanced
multiversxmultiversx
12

fix_verification

Verify smart contract bug fixes by running Mandos scenarios and regression tests.

Official
Intermediate
Oruga420Oruga420

eval-harness

Define and run capability and regression evals for Claude Code sessions.

Community
Advanced
Sunpeak-AISunpeak-AI
210

test-mcp-server

Automate MCP server testing with e2e, visual regression, live, and eval tests.

Official
Advanced