nightly-eval-investigation

What problem does it solve?

Manually reviewing cross-agent nightly evaluation results to identify broken, outdated, or over-prescribed web guidance is time-consuming and prone to missed cross-agent patterns. This Skill automates the end-to-end process of downloading, analyzing, and diagnosing issues across multiple agent runs to ensure web guidance remains effective and up-to-date.

Core Features & Use Cases

• Cross-Agent Comparison: Analyze results from Claude Code, Codex CLI, and Jetski CLI nightly runs to spot agent-agnostic issues with guidance or task design.
• Automated Health Flagging: Automatically flag 4 key problem types: missing expected guides, excessive guide consumption, low guided pass rates, and high unguided pass rates.
• Qualitative Root-Cause Analysis: Inspect task prompts, guides, and graders for flagged tasks to generate exact, actionable fix recommendations, with structured Markdown and JSON reports for downstream automation.
• Use Case: If a new View Transitions guide is causing all agents to fail nightly tasks, this Skill will flag the guide, extract common failed assertions across agents, and recommend precise changes to the prompt, guide, or grader.

Quick Start

Use the nightly-eval-investigation skill to download the latest three nightly evaluation runs, flag unhealthy tasks and guides, and generate a full diagnostic report with actionable recommendations.