eval-harness

Define, execute, and review AI model evaluations with automated reporting.

3|Updated Apr 9, 2026

Search Tags:#automation #testing #evaluation #regression #assessment #machine-learning

Ready to run

npx skills add https://github.com/rlagycks/oh-my-forge --skill eval-harness-rlagycks

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/rlagycks/oh-my-forge/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/rlagycks/oh-my-forge --skill eval-harness-rlagycks

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies the process of designing, executing, and analyzing evaluations for AI models, enabling reliable testing and regression tracking.

Core Features & Use Cases

Evaluation Framework Setup: Define capability and regression tests for AI models before implementation.
Automated Running & Reporting: Execute tests automatically and generate comprehensive evaluation reports.
Use Case: Imagine a developer adds a new feature to an AI assistant; they can quickly create evals to verify the feature's performance and detect regressions, ensuring code quality.

Quick Start

Describe your evaluation criteria to the AI system and run the check command to verify model performance.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up automated AI model evaluation for regression testing?▼

Automated AI model evaluation for regression testing requires defining capability and regression tests before implementation, then executing them automatically to generate comprehensive assessment reports. This Skill facilitates that cycle by allowing developers to define criteria, run checks, and review results aligned with development workflows.

What is the best way to detect regressions when adding new features to a machine learning model?▼

Detecting regressions in a machine learning model involves running systematic pass/fail evaluations after new features are added. You can define specific evaluation criteria aligned with your development workflow, execute the tests automatically, and review the generated reports to verify performance and ensure code quality.

Can I benchmark AI assistant performance without writing custom test scripts?▼

Benchmarking AI assistant performance involves describing your evaluation criteria to the system and running a check command to verify model behavior. This Skill provides the scripting components and references needed to execute standardized pass/fail metrics, reducing the need for extensive custom test script development.

Does this evaluation framework support performance benchmarking for machine learning projects?▼

This evaluation framework supports performance benchmarking for machine learning projects by enabling users to define, execute, and review comprehensive assessments. It applies directly to benchmarking tasks, regression detection, and feature validation by automating the pass/fail metrics generation process.

How do I define and execute capability tests before implementing an AI feature?▼

Defining and executing capability tests before implementation involves establishing pass/fail evaluation criteria aligned with development workflows. You describe the evaluation requirements to the AI system, run the automated checks, and receive comprehensive reporting to validate the intended feature performance.