Eval Analyzer

Evaluation baseline analysis automatically categorizes errors into compile_error, logic_error, and runtime_error types, then identifies top failing benchmarks. Run the analyze_failures.sh script on your eval results directory to generate a structured report pinpointing which benchmarks and models failed and why, enabling faster root-cause investigation.

Yes. The compare_models.sh script generates side-by-side success rates, costs, and token usage comparisons across models in your baseline results. This data-driven comparison helps you select the best model for your use case based on concrete performance and efficiency metrics.

Prompt accuracy verification detects mismatches between prompt documentation and actual language behavior by analyzing eval failures across benchmarks and languages. The Skill flags these discrepancies in structured outputs, catching critical prompt bugs before they compound performance issues in production.

The quick_summary.sh script generates a high-level overview of your baseline results, showing aggregate failure counts and model performance snapshots. Use this for rapid assessment before diving into detailed failure categorization or model comparisons.

Evaluation analysis produces JSONL and JSON outputs consumable by jq and shell workflows, enabling downstream automation with eval-matrix, eval-analyze, and eval-summary commands. All outputs include robust error handling and version-aware structure for reliable pipeline integration.

Run evaluation analysis immediately after executing eval baselines to understand failure patterns before iterating on prompts or language implementations. This positions analysis as a post-evaluation step that generates actionable insights for prompt refinement and model selection decisions.