skills.rest
nazrulsoftwaredev
nazrulsoftwaredevCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Define and run evaluation suites for AI-assisted development sessions.

Updated Jul 8, 2026
Search Tags:#automation#testing#benchmarking#reliability#regression#evals
One-click install
npx skills add https://github.com/nazrulsoftwaredev/NIT_CRM_2 --skill eval-harness-nazrulsoftwaredev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/nazrulsoftwaredev/NIT_CRM_2/tree/main/.agents/.agents/skills/eval-harness
Command: npx skills add https://github.com/nazrulsoftwaredev/NIT_CRM_2 --skill eval-harness-nazrulsoftwaredev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill addresses the lack of standardized testing in AI-assisted development by providing a structured framework to define, execute, and report on evaluation criteria for AI-generated code.

Core Features & Use Cases

  • Eval-Driven Development (EDD): Define pass/fail criteria before writing code to ensure objective success metrics.
  • Multi-Modal Grading: Supports deterministic code-based checks, model-based evaluation for open-ended tasks, and human review workflows.
  • Reliability Metrics: Tracks pass@k and pass^k metrics to quantify agent performance and regression stability over time.

Quick Start

Use the eval-harness skill to define a new evaluation suite for the current feature by running the eval define command followed by the feature name.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up regression testing for AI-generated code?▼

Eval-driven development formalizes AI reliability by defining pass/fail criteria before writing code, enabling deterministic checks, model-based evaluation, and human review workflows to measure agent performance objectively.

Can I track pass@k benchmarks for AI coding agents?▼

Yes, you can track pass@k and pass^k metrics to quantify agent performance and regression stability over time, integrating with project-specific test suites and Claude Code workflows to maintain consistent benchmarks.

Does the evaluation framework support model-based grading for open-ended tasks?▼

Yes, the evaluation framework supports multi-modal grading, including deterministic code-based checks, model-based evaluation for open-ended tasks, and human review workflows to ensure comprehensive reliability tracking.

What are the limitations of relying on automated grading for AI code?▼

Automated grading limitations are mitigated by combining deterministic checks with model-based evaluation and human review workflows, ensuring comprehensive reliability metrics beyond simple pass/fail criteria.

Related Skills

Explore More Software Engineering Skills→
conejoRojoconejoRojo

using-git-worktrees

Create isolated git worktrees for parallel branch development.

Community
Advanced
LincyawLincyaw

Eval Harness Skill

Implement eval-driven development with code, model, and human graders.

Community
Advanced
drgaciwdrgaciw

gitlab-ci-patterns

Generate GitLab CI/CD pipeline configurations for testing, building, and deployment.

Community
Intermediate
zero3041zero3041
1

eval-harness

Define evaluation criteria and run regression tests for Claude Code sessions.

Community
Advanced