skills.rest
dsgalkar
dsgalkarCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

benchmark-e2e

Runs end-to-end benchmark pipelines testing skill injection, dev servers, and generating improvement reports.

Updated Sep 5, 2026
Search Tags:#automation#dev server#report generation#benchmark#e2e testing#claude plugin#skill injection
One-click install
npx skills add https://github.com/dsgalkar/dnyaneshwar_portfolio --skill benchmark-e2e-dsgalkar
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark-e2e
Source: https://github.com/dsgalkar/dnyaneshwar_portfolio/tree/main/.agents/plugins/vercel/.claude/skills/benchmark-e2e
Command: npx skills add https://github.com/dsgalkar/dnyaneshwar_portfolio --skill benchmark-e2e-dsgalkar

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Verifying that a Claude plugin correctly injects skills into realistic projects requires manual, repetitive testing. This Skill automates the full loop: creating test projects, exercising skill injection via claude --print, launching dev servers, analyzing conversation logs, and producing actionable improvement reports. ## Core Features & Use Cases - Four-Stage Pipeline: Chains runner, verify, analyze, and report stages sequentially, aborting on failure with NDJSON event tracking in events.jsonl. - Precise Session Correlation: Uses run-manifest.json to link JSONL conversation sessions to projects instead of guessing from directory listings. - Machine-Readable Reports: Generates report.json with verdicts, skill injection gaps, and copy-pasteable suggestedPatterns for fixing frontmatter. - Use Case: Run the suite overnight in a loop, then wake up to timestamped reports showing which expected skills (auth, payments, ai-sdk) were never injected and exactly which glob patterns to add. ## Quick Start Run the benchmark suite with bun run scripts/benchmark-e2e.ts, optionally adding --quick to test only the first three projects.

Frequently Asked Questions about benchmark-e2e

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run an end-to-end benchmark for a Claude plugin?▼

Run bun run scripts/benchmark-e2e.ts to execute the full nine-project suite, which takes two to three hours. Add the --quick flag to run only the first three projects in roughly 30 to 45 minutes.

How do I test whether Claude skills are being injected correctly?▼

The pipeline runs claude --print with VERCEL_PLUGIN_LOG_LEVEL=trace on realistic project prompts, then matches JSONL sessions to projects via run-manifest.json. The report lists expected versus actual injected skills per project.

What does the benchmark report.json contain?▼

report.json includes a verdict of pass, partial, or fail, a gaps array listing expected skills never injected per project slug, recommendations, and suggestedPatterns with copy-pasteable glob patterns and tool names to fix frontmatter.

Can I run the benchmark suite continuously overnight?▼

Yes, wrap the command in a while loop with sleep 3600 between iterations. Each run produces timestamped report files, letting you diff verdicts across runs to confirm the trend from fail toward pass.

Why does the benchmark pipeline abort before generating a report?▼

The orchestrator chains runner, verify, analyze, and report stages sequentially and aborts on any stage failure. Check events.jsonl for the error event, which records the failed stage, exit code, and project slug.

Related Skills

Explore More Software Engineering Skills→
ChristineThamChristineTham

benchmark-testing

Create isolated benchmark test projects to exercise vercel-plugin skill injection.

Community
Advanced
dotnetdotnet
5.3k

improve-skill-quality

Diagnoses failing skill evaluations and applies targeted fixes to skills, fixtures, or eval specs.

Official
Advanced
violetiovioletio

observability

Standardize logs, metrics, and traces for distributed service monitoring.

Official
Advanced
CarlosQ96CarlosQ96

smoke-check

Run smoke tests and generate PASS/FAIL reports for QA hand-off.

Community
Intermediate