benchmark-e2e

Runs end-to-end benchmark pipelines testing skill injection across realistic projects and generates improvement reports.

Updated Aug 21, 2025
One-click install
npx skills add https://github.com/Adithiya-S/AI-Study-Companion --skill benchmark-e2e-adithiya-s
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: benchmark-e2e
Source: https://github.com/Adithiya-S/AI-Study-Companion/tree/main/.agents/skills/benchmark-e2e
Command: npx skills add https://github.com/Adithiya-S/AI-Study-Companion --skill benchmark-e2e-adithiya-s

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Verifying that a plugin correctly injects the right skills into AI coding sessions requires tedious manual testing across many project types. This Skill automates the entire loop: creating realistic test projects, running skill injection, launching dev servers, verifying they work, and producing actionable gap reports. ## Core Features & Use Cases - Four-Stage Pipeline: Chains runner, verify, analyze, and report stages sequentially, aborting on failure with NDJSON event tracking. - Precise Session Correlation: Uses a run-manifest.json contract to match JSONL conversation logs to projects instead of guessing from directory listings. - Actionable Reports: Generates report.json and report.md with verdicts, missing-skill gaps, and copy-pasteable suggestedPatterns for fixing frontmatter. - Use Case: Run the suite overnight in a loop, then wake up to timestamped reports showing which expected skills (auth, payments, ai-sdk) were never injected, letting you close gaps and watch the verdict trend from fail to pass. ## Quick Start Run the benchmark-e2e skill in quick mode to test the first three projects and show me the improvement report.

Frequently Asked Questions about benchmark-e2e

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run an end-to-end benchmark for skill injection?

Run bun run scripts/benchmark-e2e.ts to execute the full nine-project suite, or add the --quick flag to test only the first three projects in about 30-45 minutes. The pipeline runs runner, verify, analyze, and report stages sequentially.

How do I test whether a plugin injects the correct skills?

The pipeline creates realistic projects with technology-agnostic prompts, runs claude --print with trace logging, then compares expected skills against actually injected skills. The report.json gaps array lists exactly which skills were expected but never injected.

Can I run the benchmark suite overnight automatically?

Yes, wrap the command in a while loop with a sleep cooldown between iterations. Each run produces timestamped report.json and report.md files that you can diff across runs to track whether the verdict trends from fail to partial to pass.

What does the benchmark report.json contain?

The report includes a runId, a verdict of pass, partial, or fail, a gaps array listing missing skills per project slug, recommendations, and suggestedPatterns with copy-pasteable YAML globs for fixing skill frontmatter.

Why does the benchmark pipeline abort before finishing?

The orchestrator chains four stages and aborts on the first failure, recording an abort event in events.jsonl with the failed stage and exit code. Check events.jsonl to identify whether the runner, verify, analyze, or report stage failed.