eval-harness

Quantify AI-generated code quality using pass@k and pass^k metrics.

37|6|Updated Feb 14, 2026

Search Tags:#code quality #testing #llm #metrics #evaluation #pass@k

Ready to run

npx skills add https://github.com/majiayu000/vibeguard --skill eval-harness-majiayu000

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/majiayu000/vibeguard/tree/main/skills/eval-harness
Command: npx skills add https://github.com/majiayu000/vibeguard --skill eval-harness-majiayu000

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of objectively measuring the quality of AI-generated code, moving beyond simple functionality to quantifiable metrics.

Core Features & Use Cases

Quantify Code Quality: Utilizes pass@k and pass^k metrics to measure single-task success rate and overall workflow reliability.
Automated Grading: Employs various Grader types (compilation, testing, linting, coverage, code review, requirement matching, architecture) to score code.
Use Case: Improve an LLM's code generation by establishing a feedback loop where code is automatically evaluated against defined metrics, and the results inform further generation strategies.

Quick Start

Use the eval-harness skill to evaluate the quality of the generated code against the defined pass@k metrics.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is pass@k metrics for evaluating AI-generated code quality?▼

Pass@k metrics measure single-task success rate and overall workflow reliability for AI-generated code. They evaluate whether generated samples pass automated grading to establish a quantifiable feedback loop.

How do I quantify LLM code generation reliability and correctness?▼

You can quantify LLM code generation reliability by applying pass@k and pass^k metrics through automated grading. This requires integrating compilation, testing, linting, and code review tools for deterministic and probabilistic evaluation.

What automated graders can I use to evaluate code quality?▼

You can use compilation, testing, linting, coverage, code review, requirement matching, and architecture graders. These grader types automatically score AI-generated code against defined metrics to measure reliability.

Do I need testing and linting tools to evaluate code with pass@k metrics?▼

Yes, evaluating code with pass@k metrics requires integration with compilation, testing, linting, and code review tools. These tools provide the deterministic and probabilistic grading needed to score code quality.

How do I set up an automated feedback loop for LLM code generation?▼

Establish an automated feedback loop by evaluating generated code against defined pass@k metrics and using the results to inform further generation strategies. This improves LLM code reliability and correctness iteratively.

What is the difference between pass@k and pass^k metrics in code evaluation?▼

Pass@k measures single-task success rate while pass^k evaluates overall workflow reliability. Both metrics use automated grading to quantify the correctness of AI-generated code in software development workflows.