eval-model-effectiveness

To analyze pipeline observations and per-role costs for model upgrades or downgrades, run the analyze.py script. It joins observations with cost data to evaluate success rates and cost per success, generating a markdown recommendations report.

Evaluating agent model effectiveness requires projects with per-role model usage and cost data, alongside pipeline observations. The Skill joins these costs with observations to measure success rates and calculate the cost per success.

No, this Skill does not automatically change agent configurations. It produces a markdown model-recommendations report with downgrade and upgrade guidance, leaving final changes to human operators.

To generate a model recommendations report for human-in-the-loop review, execute the analyze.py script for your current project. It outputs a markdown file in your learning directory summarizing effectiveness and cost metrics.

Use an automated report to tune agent models when teams want to optimize model-tier choices like Opus, Sonnet, or Haiku based on success-rate and cost metrics. It provides data-driven downgrade and upgrade guidance for resource optimization.