skills.rest
phatblat
phatblatCOMMUNITY
Explore all skills by this author →
Website𝕏@phatblatGitHub

Evals

Evaluate AI agent workflows with code, model, and human grading.

12|Updated Aug 16, 2019
Search Tags:#quality assurance#benchmarking#llm evaluation#regression testing#ai evaluation#agent testing
One-click install
npx skills add https://github.com/phatblat/dotfiles --skill evals-phatblat
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Evals
Source: https://github.com/phatblat/dotfiles/tree/main/.claude/skills/Evals
Command: npx skills add https://github.com/phatblat/dotfiles --skill evals-phatblat

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires yaml, typescript, bun, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a robust framework for evaluating AI agent performance, ensuring quality, detecting regressions, and benchmarking capabilities against defined criteria.

Core Features & Use Cases

  • Comprehensive Evaluation: Assess agent workflows, not just single outputs, using multiple grader types (code, model, human).
  • Regression Testing: Automatically verify that agent behavior hasn't degraded after updates.
  • Capability Benchmarking: Measure and compare agent performance on specific tasks and domains.
  • Use Case: Automatically run a suite of tests against your agent after every code commit to ensure core functionalities like file manipulation and tool usage remain intact.

Quick Start

Use the Evals skill to run the 'core-behaviors' regression suite.

Frequently Asked Questions about Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run regression testing for AI agent behavior after code updates?▼

AI agent evaluation assesses agent workflows using multiple grader types, including code-based, model-based, and human grading. It measures performance metrics to ensure quality and detect regressions across defined criteria.

What is the best way to benchmark LLM evaluation capabilities for specific tasks?▼

Capability benchmarking measures and compares AI agent performance on specific tasks and domains using detailed metrics and reporting. This helps track continuous improvement and detect performance regressions.

Does this AI evaluation framework support TypeScript and Bun environments?▼

Yes, this AI evaluation framework supports TypeScript and Bun environments. It uses YAML configurations to define structured testing suites and integrates smoothly with your existing development workflow.

Can I use model-based grading for quality assurance in agent testing?▼

Yes, you can use model-based grading for quality assurance in agent testing. The framework integrates code-based, model-based, and human grading to provide comprehensive evaluation of agent workflows.

How do I automate a test suite to evaluate AI agent workflows continuously?▼

Automate a test suite to evaluate AI agent workflows by running structured tests after every code commit. This automatically verifies that agent behavior hasn't degraded and core functionalities remain intact.

Related Skills

Explore More Software Engineering Skills→
raphaelbarbosaqwertyraphaelbarbosaqwerty
7

maestro-e2e

Run Maestro end-to-end UI test flows from YAML definitions across platforms.

Community
Intermediate
ViewWayViewWay

self-improving-agent

Analyze OpenClaw conversations to identify improvement opportunities and generate weekly reports.

Community
Intermediate
nubivnubiv

eval-harness

Formalize AI model evaluation with pass@k and pass^k metrics.

Community
Advanced
roanbrasilroanbrasil
5

llm-evals

Implement LLM evaluation patterns with LangSmith, RAGAS, and PromptFoo.

Community
Advanced