Run regression and capability evaluations for multi-turn AI agent workflows.
npx skills add https://github.com/davdunc/pai-framework --skill evals-davdunc
Please help me install this Agent Skill. Skill: Evals Source: https://github.com/davdunc/pai-framework/tree/main/skills/Evals Command: npx skills add https://github.com/davdunc/pai-framework --skill evals-davdunc
The Evals skill prevents unnoticed agent quality regressions by turning messy, multi-turn agent behavior into repeatable evaluations with measurable pass rates and scores.
Ask the assistant: run evals for the Evals suite named core-behaviors and report the latest pass rate and failing tasks.
High-intent search queries and answers about installing and using this skill.
Agent regression testing prevents unnoticed quality regressions by scoring multi-turn transcripts and tool-call sequences with deterministic and model-based graders, producing measurable pass rates. It converts observed failures into new regression tasks for continuous improvement.
Pass@k measures capability by checking if an agent succeeds at least once across k trials, while pass^k measures consistency by requiring success across all k trials. Both metrics quantify multi-turn agent reliability for regression testing.
LLM-as-judge rubric graders in TypeScript require configured graders, use cases or suites, and optional Anthropic credentials for user simulation. They run alongside deterministic graders like regex and static checks for nuanced quality assessment in scenario simulation.
Yes, agent transcript scoring works with Bun, the AI SDK, and Anthropic credentials for LLM-based judging and scenario user simulation. The TypeScript environment uses YAML configuration to manage evaluation suites and compare models or prompts.
No, deterministic graders for tool call verification use regex, static checks, and tool-call sequences without needing Anthropic credentials. Anthropic credentials are only optional for LLM-based rubric judging and scenario user simulation.
The best way to convert agent evaluation failures into regression tasks is binding evaluation outcomes back into an ALGORITHM ISC pipeline. This verifies Ideal State Criteria rows and ensures the system improves over time through automated verification.
mitul-bhatiaDefine pass/fail criteria and measure AI agent performance with evaluators.
SimHackerCreate weighted rubrics and score outputs against defined criteria.
ylxaiEvaluate AI agent transcripts with code-based, model-based, and human graders.
automatedigitalEnforce regex and grammar constraints during LLM inference for valid structured outputs.
