skills.rest
majiayu000
majiayu000COMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Quantify AI-generated code quality using pass@k and pass^k metrics.

37|6|Updated Feb 14, 2026
Search Tags:#code quality#testing#llm#metrics#evaluation#pass@k
One-click install
npx skills add https://github.com/majiayu000/vibeguard --skill eval-harness-majiayu000
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/majiayu000/vibeguard/tree/main/skills/eval-harness
Command: npx skills add https://github.com/majiayu000/vibeguard --skill eval-harness-majiayu000

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of objectively measuring the quality of AI-generated code, moving beyond simple functionality to quantifiable metrics.

Core Features & Use Cases

  • Quantify Code Quality: Utilizes pass@k and pass^k metrics to measure single-task success rate and overall workflow reliability.
  • Automated Grading: Employs various Grader types (compilation, testing, linting, coverage, code review, requirement matching, architecture) to score code.
  • Use Case: Improve an LLM's code generation by establishing a feedback loop where code is automatically evaluated against defined metrics, and the results inform further generation strategies.

Quick Start

Use the eval-harness skill to evaluate the quality of the generated code against the defined pass@k metrics.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is pass@k metrics for evaluating AI-generated code quality?▼

Pass@k metrics measure single-task success rate and overall workflow reliability for AI-generated code. They evaluate whether generated samples pass automated grading to establish a quantifiable feedback loop.

How do I quantify LLM code generation reliability and correctness?▼

You can quantify LLM code generation reliability by applying pass@k and pass^k metrics through automated grading. This requires integrating compilation, testing, linting, and code review tools for deterministic and probabilistic evaluation.

What automated graders can I use to evaluate code quality?▼

You can use compilation, testing, linting, coverage, code review, requirement matching, and architecture graders. These grader types automatically score AI-generated code against defined metrics to measure reliability.

Do I need testing and linting tools to evaluate code with pass@k metrics?▼

Yes, evaluating code with pass@k metrics requires integration with compilation, testing, linting, and code review tools. These tools provide the deterministic and probabilistic grading needed to score code quality.

How do I set up an automated feedback loop for LLM code generation?▼

Establish an automated feedback loop by evaluating generated code against defined pass@k metrics and using the results to inform further generation strategies. This improves LLM code reliability and correctness iteratively.

What is the difference between pass@k and pass^k metrics in code evaluation?▼

Pass@k measures single-task success rate while pass^k evaluates overall workflow reliability. Both metrics use automated grading to quantify the correctness of AI-generated code in software development workflows.

Related Skills

Explore More Software Engineering Skills→
farmountainfarmountain
1

Build Validator Agent

Validate software builds through automated testing, linting, and quality gates.

Community
Advanced
KuaaMUKuaaMU
4

eval-harness

Implement Eval-Driven Development frameworks for AI code generation sessions.

Community
Advanced
limzivectorlimzivector

plan-eng-review

Review engineering plans and architecture for design, code quality, tests, and performance.

Community
Advanced
JamkrisJamkris
86

eval-harness

Automate eval-driven testing workflows for Gemini CLI sessions with deterministic graders.

Community
Advanced