diff-eval-func

Analyze agent-generated code changes against human-approved PRs with file and function coverage.

3|2|Updated Mar 14, 2026

Search Tags:#automation #code-quality #deterministic #patch #pr-review #diff-eval #function-level

Ready to run

npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval-func

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: diff-eval-func
Source: https://github.com/wzh4464/claude-skills/tree/main/diff-eval-func
Command: npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval-func

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluate agent-generated code changes against a human-approved PR, using deterministic file coverage and function-level coverage analysis to ensure reproducible evaluation of patch quality across repos and languages.

Core Features & Use Cases

Deterministic file coverage computed via bash scripts to ensure consistent results across runs.
Function-level coverage that identifies which functions are modified by diffs.
Dual semantic + data-based analysis with structured scoring and auto-generated file filtering.
Ground-truth vs generated patch comparison for PR validation and AI coding tool benchmarking.

Quick Start

Provide the PR diff, the ground-truth patch, and the repository path to run a deterministic, function-level diff evaluation and generate a structured evaluation report.

Frequently Asked Questions about diff-eval-func

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate agent-generated code changes against a human-approved PR?▼

To evaluate agent-generated code changes against a human-approved PR, use deterministic file coverage and function-level coverage analysis. This approach compares a PR diff with a ground-truth patch to generate a structured, reproducible evaluation report.

What is deterministic function-level diff evaluation?▼

Deterministic function-level diff evaluation is an analysis method that identifies which specific functions are modified by code changes. It computes file coverage via bash scripts to ensure consistent, reproducible patch quality results across different runs and repositories.

How do I run a function-level diff evaluation for PR validation?▼

To run a function-level diff evaluation, trigger the /diff-eval-func command and provide a PR diff, a ground-truth patch, and a repository path or patch file. The system then executes the evaluation and generates a structured report.

Does diff evaluation support both semantic and data-based analysis?▼

Yes, diff evaluation supports dual semantic and data-based analysis. This dual approach provides structured scoring and automated file filtering to comprehensively assess patch quality during PR validation and AI coding tool benchmarking.

Can I use patch evaluation for benchmarking AI coding tools across different repos?▼

Yes, you can use patch evaluation for benchmarking AI coding tools across multiple repositories and programming languages. It performs ground-truth versus generated patch comparison to ensure reproducible evaluation of patch quality.

What are the limitations of deterministic file coverage in patch evaluation?▼

Deterministic file coverage relies on bash scripts for consistent results, which may require a compatible environment. The evaluation specifically requires a ground-truth patch and a PR diff to function, limiting its use for patches without a baseline reference.