skills.rest
hector-manny
hector-mannyCOMMUNITY
Explore all skills by this author →
Website𝕏@Hector_MrtzGitHub

eval-harness

Implement eval-driven development frameworks with pass@k metrics for AI code.

Updated Jun 22, 2026
Search Tags:#code review#ai development#ai testing#evaluation#edd
One-click install
npx skills add https://github.com/hector-manny/bussbot --skill eval-harness-hector-manny
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/hector-manny/bussbot/tree/main/.cursor/.agents/skills/eval-harness
Command: npx skills add https://github.com/hector-manny/bussbot --skill eval-harness-hector-manny

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI code and development practices, ensuring quality and reliability.

Core Features & Use Cases

  • Formal Evaluation Framework: Implement eval-driven development (EDD) principles.
  • Eval Types: Capability and regression evaluations to ensure functionality and reliability.
  • Grader Types: Code-based, model-based, and human graders for comprehensive assessments.
  • Metrics: Track pass@k metrics for reliability measurement.
  • Integration Patterns: Pre-implementation, during implementation, and post-implementation support.
  • Eval Storage: Organized storage of evals for version control and history tracking.
  • Best Practices: Guidance for best practices in eval-driven development.

Quick Start

To begin defining an evaluation, use the command: /eval define feature-name.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development and how does it evaluate AI code?▼

Eval-driven development (EDD) is a formal evaluation framework for AI code that uses capability and regression evaluations with code-based, model-based, and human graders to ensure functionality and reliability.

How do I define an evaluation for an AI feature?▼

To define an evaluation for an AI feature, use the command `/eval define feature-name` to establish structured evaluation definitions that integrate directly into your development workflow.

What metrics are used to measure AI code reliability during evaluation?▼

AI code reliability is measured using pass@k metrics, which track the consistency and success rate of your capability and regression evaluations across different grader types.

When should I integrate evaluations into my AI development workflow?▼

Evaluations can be integrated pre-implementation, during implementation, and post-implementation, providing continuous quality and reliability tracking throughout the entire AI development lifecycle.

Does eval-driven development support version control for evaluation definitions?▼

Yes, eval-driven development provides organized storage of evaluations, enabling version control and history tracking for structured evaluation definitions used in capability and regression testing.

What types of graders work with formal AI code evaluation frameworks?▼

Formal AI code evaluation frameworks support three grader types: code-based graders, model-based graders, and human graders, ensuring comprehensive assessments for capability and regression evaluations.

Related Skills

Explore More Software Engineering Skills→
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
thanhquan3010thanhquan3010

eval-harness

Define capability and regression evaluations with code, model, or human graders.

Community
Advanced
fideguchfideguch
1

eval-harness

Define and run capability and regression evals for Claude Code sessions.

Community
Advanced
ryasrkryasrk

eval-harness

Formalize evaluation frameworks for AI code sessions with pass@k and pass^k metrics.

Community
Advanced