eval-check

Run deterministic and LLM-based agent quality evaluations after code changes.

Updated Feb 2, 2026

Search Tags:#code quality #testing #llm #agent #evaluation #regression

Ready to run

npx skills add https://github.com/rosinbum/usopc-athlete-support-agent --skill eval-check

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-check
Source: https://github.com/rosinbum/usopc-athlete-support-agent/tree/main/.claude/skills/eval-check
Command: npx skills add https://github.com/rosinbum/usopc-athlete-support-agent --skill eval-check

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pnpm, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill ensures the AI agent's quality and correctness by automatically running evaluations after code changes, preventing regressions and maintaining performance.

Core Features & Use Cases

Automated Code Change Detection: Identifies modified agent files across core, tools, and services.
Tiered Evaluation Execution: Runs fast, deterministic tests first, then prompts for confirmation before executing expensive LLM-based evaluations.
Selective Evals: Allows running only relevant LLM evaluations based on the type of code changes detected.
Use Case: After a developer modifies the agent's prompt templates, this skill runs evaluations to confirm that the agent's responses remain grounded, correct, and properly cited.

Quick Start

Run agent quality evaluations after core agent code changes.

Frequently Asked Questions about eval-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run agent quality evaluations after modifying core agent code?▼

Agent quality evaluations run automatically after code modifications by executing deterministic checks first, then conditionally running LLM-based tests for groundedness, correctness, trajectory, citations, and disclaimers.

What is tiered evaluation execution for LLM agent testing?▼

Tiered evaluation execution runs fast, deterministic tests first, then prompts for confirmation before executing expensive LLM-based evaluations. This mechanism optimizes testing resources by validating basic correctness before committing to costly LLM inference checks.

Can I run selective LLM evaluations based on specific code changes?▼

Yes, selective evaluations allow running only relevant LLM tests based on the type of code changes detected. The skill categorizes modifications in core agent packages and selects matching evaluations to verify groundedness, trajectory, and citations.

Do I need pnpm to automate regression testing for AI agents?▼

Yes, pnpm is required as a dependency to automate the execution of agent quality evaluations triggered by code modifications. This package manager orchestrates the scripts that run deterministic and LLM-based tests to prevent agent performance regressions.

How does automated code change detection work for agent regression prevention?▼

Automated code change detection identifies modified agent files across core, tools, and services. It categorizes these changes and selects relevant evaluations, running deterministic checks before conditionally executing LLM-based tests to prevent regressions.

What are the limitations of running LLM-based agent evaluations?▼

LLM-based agent evaluations are expensive and require confirmation before execution. The skill mitigates this by running fast deterministic tests first and only conditionally running LLM tests for groundedness, correctness, trajectory, citations, and disclaimers based on detected changes.