diff-eval-local

Evaluate agent-generated code against ground-truth diffs and handwritten file lists.

3|2|Updated Mar 14, 2026

Search Tags:#agent #evaluation #code #patch #ground-truth #handwritten #diff-eval

Ready to run

npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval-local

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: diff-eval-local
Source: https://github.com/wzh4464/claude-skills/tree/main/diff-eval-local
Command: npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval-local

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Evaluate agent-generated code against a ground-truth diff and a handwritten file list to enable reproducible scoring and analysis.

Core Features & Use Cases

Deterministic file-coverage analysis comparing handwritten HW files to generated changes and repo modifications.
GT-diff integration: reads ground-truth patches from the base_repo task to anchor evaluation.
Function-level context extraction and metadata-driven prompt resolution to guide execution and scoring.

Quick Start

Invoke /diff-eval-local with your experiment repository path and the corresponding ground-truth diff and handwritten file list to produce a deterministic evaluation report.

Frequently Asked Questions about diff-eval-local

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate agent-generated code against a ground-truth diff?▼

To evaluate agent-generated code, compare it against a ground-truth diff and a handwritten file list. This process automates deterministic file coverage analysis and parses GT diffs to anchor scoring, enabling reproducible analysis.

What is deterministic file coverage analysis for code patches?▼

Deterministic file coverage analysis is a method comparing handwritten files to generated changes and repo modifications. It provides reproducible scoring by anchoring evaluation to a ground-truth diff.

How do I extract function-level context from a ground-truth diff?▼

Function-level context extraction reads ground-truth patches from the base repository task. It uses a metadata-driven prompt resolution workflow to guide execution and scoring of proposed code changes.

Can I score agent code changes using a handwritten file list?▼

Yes, you can score agent code changes using a handwritten file list combined with a GT diff. This approach ensures deterministic evaluation by comparing generated modifications against expected ground-truth files.

Does agent code evaluation require a base repository?▼

Yes, agent code evaluation requires a base repository to anchor ground-truth patches. The evaluation reads these GT diffs from the base repository task to accurately parse and score proposed modifications.

What are the limitations of using GT diffs for code evaluation?▼

Using GT diffs limits evaluation to changes present in the ground-truth patch. It requires a predefined handwritten file list and base repository, meaning unlisted modifications or structural deviations may not be scored accurately.