Evals

Evaluate AI agent workflows with code, model, and human grading.

12|Updated Aug 16, 2019

Search Tags:#quality assurance #benchmarking #llm evaluation #regression testing #ai evaluation #agent testing

Ready to run

npx skills add https://github.com/phatblat/dotfiles --skill evals-phatblat

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Evals
Source: https://github.com/phatblat/dotfiles/tree/main/.claude/skills/Evals
Command: npx skills add https://github.com/phatblat/dotfiles --skill evals-phatblat

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires yaml, typescript, bun, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a robust framework for evaluating AI agent performance, ensuring quality, detecting regressions, and benchmarking capabilities against defined criteria.

Core Features & Use Cases

Comprehensive Evaluation: Assess agent workflows, not just single outputs, using multiple grader types (code, model, human).
Regression Testing: Automatically verify that agent behavior hasn't degraded after updates.
Capability Benchmarking: Measure and compare agent performance on specific tasks and domains.
Use Case: Automatically run a suite of tests against your agent after every code commit to ensure core functionalities like file manipulation and tool usage remain intact.

Quick Start

Use the Evals skill to run the 'core-behaviors' regression suite.

Frequently Asked Questions about Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run regression testing for AI agent behavior after code updates?▼

AI agent evaluation assesses agent workflows using multiple grader types, including code-based, model-based, and human grading. It measures performance metrics to ensure quality and detect regressions across defined criteria.

What is the best way to benchmark LLM evaluation capabilities for specific tasks?▼

Capability benchmarking measures and compares AI agent performance on specific tasks and domains using detailed metrics and reporting. This helps track continuous improvement and detect performance regressions.

Does this AI evaluation framework support TypeScript and Bun environments?▼

Yes, this AI evaluation framework supports TypeScript and Bun environments. It uses YAML configurations to define structured testing suites and integrates smoothly with your existing development workflow.

Can I use model-based grading for quality assurance in agent testing?▼

Yes, you can use model-based grading for quality assurance in agent testing. The framework integrates code-based, model-based, and human grading to provide comprehensive evaluation of agent workflows.

How do I automate a test suite to evaluate AI agent workflows continuously?▼

Automate a test suite to evaluate AI agent workflows by running structured tests after every code commit. This automatically verifies that agent behavior hasn't degraded and core functionalities remain intact.