benchmark-agents

Runs interactive Claude Code eval sessions to verify Vercel plugin skill injection and hook behavior.

Updated Aug 21, 2025
One-click install
npx skills add https://github.com/Adithiya-S/AI-Study-Companion --skill benchmark-agents-adithiya-s
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: benchmark-agents
Source: https://github.com/Adithiya-S/AI-Study-Companion/tree/main/.agents/skills/benchmark-agents
Command: npx skills add https://github.com/Adithiya-S/AI-Study-Companion --skill benchmark-agents-adithiya-s

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Testing a Claude Code plugin's skill injection and hook behavior requires real interactive sessions, since print-mode runs never fire hooks or create files. This Skill provides the exact commands, monitoring queries, and verification checks to run reliable end-to-end evals of the Vercel plugin. ## Core Features & Use Cases - Interactive Eval Launch: Spawns WezTerm panes running real Claude Code sessions with the plugin installed via npx add-plugin, using timestamped project directories. - Injection Monitoring: Checks skill claim directories and debug logs to count injected skills, hook firings, and PostToolUse validation catches. - Code Verification: Greps generated projects for correct patterns such as withWorkflow config, AI Gateway usage, Gemini image models, and AI Elements components. - Use Case: After updating a skill's prompt signals, launch three parallel eval sessions with natural-language prompts, confirm the expected skills were injected, inspect the generated Next.js code, then write a coverage report before releasing. ## Quick Start Launch a benchmark eval by creating a timestamped directory under ~/dev/vercel-plugin-testing, installing the Vercel plugin with npx add-plugin, and spawning a WezTerm pane running an interactive Claude session with a natural-language product prompt.

Frequently Asked Questions about benchmark-agents

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test Claude Code plugin skill injection?

Install the plugin with npx add-plugin in a test directory, then spawn an interactive Claude session via wezterm cli spawn with VERCEL_PLUGIN_LOG_LEVEL=debug. Check the session's claim directory in the temp folder to see which skills were injected.

Why doesn't claude --print work for plugin evals?

Print mode generates text without executing tools, so PreToolUse, PostToolUse, and UserPromptSubmit hooks never fire. No files are created, no dependencies install, and no session_id exists for claim files or debug logs.

How do I check which skills were injected in a session?

List the contents of the claim directory at $TMPDIR/vercel-plugin-<session-id>-seen-skills.d. Each file represents one injected skill, and you can count them or grep for specific skills like workflow.

What prompts trigger the workflow skill in evals?

Use natural product language like multi-step pipeline, streams progress, or durable pipeline rather than naming technologies. The plugin's promptSignals match these phrases to inject the workflow skill.

Why must eval directories avoid uppercase letters?

npm rejects uppercase letters in package names, so a capital letter in a timestamped directory name breaks create-next-app scaffolding. Use lowercase slugs with date-based suffixes like my-app-20260309-1227.