model-bakeoff

What problem does it solve?

This Skill addresses the challenge of comparing the performance of multiple models on the same task, ensuring a fair and thorough evaluation.

Core Features & Use Cases

• Multi-Model Comparison: Conduct controlled bake-offs among various models to compare their performance.
• Task-Based Evaluation: Ensure models diagnose, plan, execute, and score the same change in isolated environments.
• Objective & Subjective Scoring: Use both objective metrics (e.g., typecheck, build, test) and subjective evaluation (code quality, intent fidelity) to score model performance.
• Merge & Grafting: Merge the best outcomes from each model, while also incorporating improvements from other models.

Quick Start

To initiate a model bake-off, use the command: `model-bakeoff run --models "opus", "sonnet", "gpt-5.5" --task "refactor feature X" --branch "experiment" --judge "Fable" --metric "typecheck + build + test + betterer" --threshold 80%.