Evals

Run regression and capability evaluations for multi-turn AI agent workflows.

1|Updated Apr 16, 2026

Search Tags:#regression testing #pass@k #LLM-as-judge #scenario simulation #agent evaluation #tool call verification #transcript scoring

Ready to run

npx skills add https://github.com/davdunc/pai-framework --skill evals-davdunc

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Evals
Source: https://github.com/davdunc/pai-framework/tree/main/skills/Evals
Command: npx skills add https://github.com/davdunc/pai-framework --skill evals-davdunc

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bun, @ai-sdk/anthropic, @langwatch/scenario, ai, yaml, typescript, and includes scripts (resource) and references (resource) components.

What problem does it solve?

The Evals skill prevents unnoticed agent quality regressions by turning messy, multi-turn agent behavior into repeatable evaluations with measurable pass rates and scores.

Core Features & Use Cases

Workflow-driven agent evaluation: Run eval suites, compare models/prompts, create judges, and generate scenarios for multi-turn transcript testing.
Grader diversity for different failure modes: Use fast deterministic graders (regex/tool calls/static checks) alongside LLM-based rubric graders for nuanced quality assessment.
Pass@k and pass^k reliability metrics: Measure capability (can it succeed at least once) and consistency (does it succeed across trials).
ALGORITHM ISC verification integration: Verify Ideal State Criteria rows by binding evaluation outcomes back into THE ALGORITHM pipeline.
Failure-to-task conversion: Convert observed failures into new regression tasks so the system improves over time.

Quick Start

Ask the assistant: run evals for the Evals suite named core-behaviors and report the latest pass rate and failing tasks.

Frequently Asked Questions about Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run regression testing for multi-turn AI agent workflows to prevent quality regressions?▼

Agent regression testing prevents unnoticed quality regressions by scoring multi-turn transcripts and tool-call sequences with deterministic and model-based graders, producing measurable pass rates. It converts observed failures into new regression tasks for continuous improvement.

What is the difference between pass@k and pass^k metrics in LLM evaluation?▼

Pass@k measures capability by checking if an agent succeeds at least once across k trials, while pass^k measures consistency by requiring success across all k trials. Both metrics quantify multi-turn agent reliability for regression testing.

How do I set up LLM-as-judge rubrics for scenario simulation in TypeScript?▼

LLM-as-judge rubric graders in TypeScript require configured graders, use cases or suites, and optional Anthropic credentials for user simulation. They run alongside deterministic graders like regex and static checks for nuanced quality assessment in scenario simulation.

Can I use Bun and the AI SDK with Anthropic for agent transcript scoring?▼

Yes, agent transcript scoring works with Bun, the AI SDK, and Anthropic credentials for LLM-based judging and scenario user simulation. The TypeScript environment uses YAML configuration to manage evaluation suites and compare models or prompts.

Do I need Anthropic API credentials to run deterministic graders for tool call verification?▼

No, deterministic graders for tool call verification use regex, static checks, and tool-call sequences without needing Anthropic credentials. Anthropic credentials are only optional for LLM-based rubric judging and scenario user simulation.

What's the best way to convert agent evaluation failures into regression tasks?▼

The best way to convert agent evaluation failures into regression tasks is binding evaluation outcomes back into an ALGORITHM ISC pipeline. This verifies Ideal State Criteria rows and ensures the system improves over time through automated verification.