skills.rest
SeanChenR
SeanChenRCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Automate evaluation and regression tracking for AI-assisted coding sessions.

Updated Apr 21, 2026
Search Tags:#evaluation#prompt-engineering#eval#regression-tests#pass@k#graders#qa-pipeline
One-click install
npx skills add https://github.com/SeanChenR/ghibli --skill eval-harness-seanchenr
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/SeanChenR/ghibli/tree/main/.claude/skills/eval-harness
Command: npx skills add https://github.com/SeanChenR/ghibli --skill eval-harness-seanchenr

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates evaluation and reliability verification for AI-assisted coding sessions.

Core Features & Use Cases

  • Eval framework templates: Predefine capability and regression tests to quantify AI performance.
  • Multi-grader support: Deterministic code graders, model-based graders, and human review to cover diverse outputs.
  • Regression tracking & baselines: Maintain baselines across model versions and monitor pass@k metrics over time.
  • Use Case: Teams can integrate this into their CI pipelines to gate AI changes before release.

Quick Start

Define an eval suite, implement tests, run the evaluation pipeline, and review the resulting report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up regression tests for AI-assisted coding sessions?▼

Regression tests for AI-assisted coding sessions are set up by defining an evaluation suite with predefined pass/fail criteria, implementing the tests, running the evaluation pipeline, and reviewing the resulting report to track regressions across model versions.

What is a pass@k metric in prompt engineering evaluation?▼

Pass@k is an evaluation metric used in prompt engineering to quantify AI performance by measuring the probability that at least one of k generated outputs passes predefined capability or regression tests within the evaluation framework.

Can I use both deterministic and model-based graders in an AI evaluation pipeline?▼

Yes, the AI evaluation pipeline supports multi-grader configurations, allowing you to apply deterministic code graders, model-based graders, and human review simultaneously to comprehensively evaluate diverse AI task outputs.

How do I track AI model regressions across different versions?▼

AI model regressions are tracked across different versions by maintaining baselines within the evaluation framework, running regression tests against new model versions, and monitoring pass@k metrics over time to identify performance degradations.

Does formal AI evaluation work with CI pipelines for gating releases?▼

Yes, formal AI evaluation works with CI pipelines by integrating the evaluation framework to run capability and regression tests, ensuring that AI changes must pass predefined pass/fail criteria before being approved for release.

Related Skills

Explore More Software Engineering Skills→
maltemdmaltemd

designing-evaluations-for-agents

Design evaluation frameworks with metrics and pass/fail thresholds for LLM-based agents.

Community
Advanced
kitfunsokitfunso

eval-driven-dev

Turn AI development evaluations into mandatory unit tests with pass@k metrics.

Community
Intermediate
JamkrisJamkris
86

eval-harness

Automate eval-driven testing workflows for Gemini CLI sessions with deterministic graders.

Community
Advanced
as4584as4584
2

llm-evaluation

Evaluate LLM applications with automated metrics and human feedback.

Community
Advanced