Compare generated patches to ground-truth PR diffs and report scoring.
npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval
Please help me install this Agent Skill. Skill: diff-eval Source: https://github.com/wzh4464/claude-skills/tree/main/diff-eval Command: npx skills add https://github.com/wzh4464/claude-skills --skill diff-eval
Evaluates agent-generated code changes against human-approved PRs using a formal rubric, diff analysis, and auto-generated file filtering to ensure alignment with ground truth.
Compute the base commit, derive the generated patch, compare it against the PR diff, and generate a full evaluation report.
High-intent search queries and answers about installing and using this skill.
You can evaluate AI-generated patches by comparing them against ground-truth PR diffs while filtering out auto-generated files. This process produces a structured evaluation report detailing data-based and semantic coverage against a formal scoring rubric.
Structured diff comparison is the process of comparing local agent changes relative to a PR base commit with the ground-truth PR diff. It performs data-based coverage on file-level changes and semantic checks on functional intent to ensure alignment with ground truth.
Score code generation tools by determining the PR base commit and head, computing the generated patch, and comparing it to the ground-truth PR diff. This produces a formal evaluation report based on functional correctness, completeness, and behavioral equivalence.
Yes, the evaluation excludes auto-generated files from the PR diff analysis. It compares the generated patch to the ground-truth PR diff while actively filtering out auto-generated files to ensure accurate measurement of functional intent and data-based coverage.
Yes, you can use diff analysis for benchmarking code generation tools by comparing their output patches against human-approved PRs. The evaluation artifacts the results for audits, measuring functional correctness and behavioral equivalence against a formal rubric.
Semantic checks in patch evaluation are limited to verifying functional intent and behavioral equivalence against the ground-truth PR diff. The evaluation relies on comparing computed patches relative to the base commit and does not account for changes outside the defined PR scope.
wzh4464Analyze agent-generated code changes against human-approved PRs with file and function coverage.
anthropicsAudit Anthropic Cookbook notebooks with rubric scoring and automated validation checks.
Quill-AgentAutomate code analysis, generation, and refactoring via the Claude Code CLI.
niznik-devCreate custom inspect-ai evaluation tasks with experiment-guided and standalone modes.
