review-fix-bench

To benchmark code-reviewer agent prompts, execute parallel test runs against ground-truth fixtures and calculate comparative performance metrics like precision, recall, and F1 scores to verify if prompt changes actually improve agent performance.

Semantic evaluation uses an LLM judge agent to compare code-reviewer outputs against defined ground-truth test fixtures, generating precision, recall, and F1 score reports to quantify whether a new prompt version outperforms the previous one.

Yes, git is required to run these A/B benchmarks, allowing the skill to compare your current code-reviewer agent prompt against previous versions stored in git history for iterative prompt refinement.

Before A/B testing code-review agent prompts, you must configure an LLM judge agent for semantic evaluation and define a set of ground-truth test fixtures to serve as the baseline for calculating precision, recall, and F1 scores.

Yes, you can use this benchmarking skill to ensure that a refined code-reviewer agent prompt improves bug detection accuracy without introducing false positives by comparing its metrics against the previous prompt version.

The limitation of using automated metrics for prompt engineering is that the evaluation relies entirely on the accuracy of the judge agent and the quality of the defined ground-truth test fixtures used to calculate the comparative performance reports.