eval-harness

Define and run end-to-end evaluations for Claude Code sessions with pass@k metrics.

Updated Mar 18, 2026

Search Tags:#ai #coding #test #benchmarks #grading #eval #evaluation-framework

Ready to run

npx skills add https://github.com/ignidus/iscagent --skill eval-harness-ignidus

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ignidus/iscagent/tree/main/skills/eval-harness
Command: npx skills add https://github.com/ignidus/iscagent --skill eval-harness-ignidus

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Eval-driven development for AI-assisted coding by providing a repeatable, objective framework to define, execute, and track evaluations across model changes and code tasks.

Core Features & Use Cases

Structured evaluation templates for capability and regression tests
Multi-grader support (Code-Based Grader, Model-Based Grader, and Human Grader) with pass@k and pass^k metrics
Baseline management, eval storage, and release-ready reporting
Integrations with CI-like workflows and project audits

Quick Start

Create and run your first eval using the provided templates and definitions.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is eval-driven development for AI-assisted coding?▼

It uses structured evaluation templates for capability and regression tests, supporting multi-grader setups including Code-Based, Model-Based, and Human Graders to measure pass@k metrics.

How do I set up an evaluation framework for AI coding sessions?▼

The framework supports capability and regression evals with measurable pass@k and pass^k metrics across code changes and prompts, integrating with CI-like workflows and project audits.

Can I use model-based and human graders together for code generation benchmarks?▼

This multi-grader approach allows you to combine deterministic grading with model rubrics and human evaluation to track run history and release-ready evaluations.

Does this evaluation framework support CI-like workflows and baseline management?▼

It implements storage patterns to track baselines, run history, and release-ready evaluations, enabling repeatable and objective assessment of AI coding capabilities.

What are the limitations of using a formal eval framework for AI coding?▼

Without proper baseline management and eval storage patterns, the framework cannot effectively track run history or determine release-ready evaluations across model changes.