skills.rest
bbaserdem
bbaserdemCOMMUNITY
Explore all skills by this author →
𝕏@bbaserdemGitHub

eval-harness

Implement eval-driven development with capability and regression evals for AI code generation.

2|Updated Apr 1, 2024
Search Tags:#code quality#testing#devops#ai#evaluation#reliability
One-click install
npx skills add https://github.com/bbaserdem/NixOS-Config --skill eval-harness-bbaserdem
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/bbaserdem/NixOS-Config/tree/main/home-manager/batuhan/apps/claude/skills/eval-harness
Command: npx skills add https://github.com/bbaserdem/NixOS-Config --skill eval-harness-bbaserdem

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI code generation sessions, ensuring reliability and tracking progress through formal evaluation (eval) driven development.

Core Features & Use Cases

  • Eval-Driven Development: Implements principles where expected behavior is defined before implementation, and evals are run continuously.
  • Eval Types: Supports Capability Evals (testing new features) and Regression Evals (ensuring existing functionality isn't broken).
  • Grader Types: Integrates Code-Based Graders (deterministic checks), Model-Based Graders (AI evaluation), and Human Graders (manual review flags).
  • Metrics: Tracks pass@k and pass^k metrics for reliability measurement.
  • Use Case: Before writing a new authentication module, define capability evals for user registration and login, and regression evals to ensure existing routes remain accessible. Run these evals throughout development to guarantee correctness.

Quick Start

Use the eval-harness skill to define a new capability eval for the 'user-registration' feature.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I formalize AI code generation evaluation with eval-driven development?▼

AI code generation evaluation is formalized by defining expected behavior before implementation and running continuous capability and regression evals to track pass@k and pass^k reliability metrics.

What is the difference between capability evals and regression evals?▼

Capability evals test new features, while regression evals ensure existing functionality is not broken. Both types are required before implementation to maintain structured development during AI code generation.

How do I set up code-based and model-based graders for AI evaluation?▼

AI evaluation graders are set up by integrating code-based deterministic checks, model-based AI evaluation, and human grader manual review flags to measure pass@k and pass^k reliability metrics during development.

Does eval-driven development work without pre-implementation definitions?▼

Eval-driven development requires pre-implementation definitions. Expected behavior must be defined before coding begins, and continuous evaluation is facilitated throughout the session to guarantee correctness.

What are pass@k and pass^k reliability metrics in AI code evaluation?▼

Pass@k and pass^k reliability metrics measure the consistency and correctness of AI code generation. They are tracked by the evaluation framework during continuous capability and regression testing.

Related Skills

Explore More Software Engineering Skills→
udecodeudecode
1.2k

test-driven-development

Enforces test-first development by writing failing tests, implementing minimal code, and refactoring.

Official
Basic
LincyawLincyaw

Eval Harness Skill

Implement eval-driven development with code, model, and human graders.

Community
Advanced
Clean1inesClean1ines

eval-harness

Evaluate Claude Code sessions with configurable graders and run pipelines.

Community
Advanced
z1439527767z1439527767

agent-evaluation

Evaluate and benchmark LLM agents for reliability and behavioral quality.

Community
Advanced