What problem does it solve?
Project Marlin provides a structured, repeatable workflow for evaluating AI-generated Python code changes on real GitHub repositories. It guides you through assembling prompts, selecting meaningful PRs, running a three-turn evaluation, and rating and justifying outcomes, all while ensuring decisions are grounded in your own analysis and not external models.
Core Features & Use Cases
- Six-step workflow: Assessment, PR selection, prompt preparation, CLI setup, PR creation, and submission, with strict sequential ordering.
- Three-turn evaluation: Execute at least three deliberate turns (A vs B) and justify the better response.
- PR-context-driven prompts: Create task-focused prompts that describe outcomes and behaviors, not implementation details.
- Non-PR references: Avoid relying on existing PRs; work from a pre-PR baseline for reproducibility.
- Production readiness mindset: Emphasize edge cases, acceptance criteria, and safe, auditable changes.
Quick Start
Follow Marlin's six-step workflow to evaluate an AI-generated PR on a real repository, starting with expert assessment and ending with submission.