Evaluate agent-generated code against ground-truth diffs and handwritten file lists.
npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval-local
Please help me install this Agent Skill. Skill: diff-eval-local Source: https://github.com/wzh4464/claude-skills/tree/main/diff-eval-local Command: npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval-local
Evaluate agent-generated code against a ground-truth diff and a handwritten file list to enable reproducible scoring and analysis.
Invoke /diff-eval-local with your experiment repository path and the corresponding ground-truth diff and handwritten file list to produce a deterministic evaluation report.
High-intent search queries and answers about installing and using this skill.
To evaluate agent-generated code, compare it against a ground-truth diff and a handwritten file list. This process automates deterministic file coverage analysis and parses GT diffs to anchor scoring, enabling reproducible analysis.
Deterministic file coverage analysis is a method comparing handwritten files to generated changes and repo modifications. It provides reproducible scoring by anchoring evaluation to a ground-truth diff.
Function-level context extraction reads ground-truth patches from the base repository task. It uses a metadata-driven prompt resolution workflow to guide execution and scoring of proposed code changes.
Yes, you can score agent code changes using a handwritten file list combined with a GT diff. This approach ensures deterministic evaluation by comparing generated modifications against expected ground-truth files.
Yes, agent code evaluation requires a base repository to anchor ground-truth patches. The evaluation reads these GT diffs from the base repository task to accurately parse and score proposed modifications.
Using GT diffs limits evaluation to changes present in the ground-truth patch. It requires a predefined handwritten file list and base repository, meaning unlisted modifications or structural deviations may not be scored accurately.
wzh4464Analyze agent-generated code changes against human-approved PRs with file and function coverage.
slowman2084Analyze prompt histories to detect anti-patterns and generate retrospective reports.
yiwei79Evaluate and refine agent outputs through structured reflection loops.
imehrGuides unit, integration, and end-to-end testing with behavior-focused rules and TDD workflows.
