benchmark-e2e

End-to-end benchmark automation orchestrates test project creation, skill injections, dev-server validation, and results analysis to generate actionable reports. This skill executes the full benchmark suite automatically to capture performance and behavior discrepancies.

An end-to-end benchmark pipeline systematically evaluates plugins by creating test projects, launching dev servers, and verifying outcomes. It generates structured metrics like run-manifest.json, events.jsonl, and report.json to track progress and expose gaps for continuous improvement.

Yes, end-to-end benchmark automation is designed specifically for CI pipelines and nightly self-improvement loops. It ensures end-to-end orchestration, manifest generation, and traceable events for auditability to produce actionable improvement reports automatically.

Generating structured metrics for plugin performance testing requires running the benchmark suite which outputs run-manifest.json, events.jsonl, and report.json files. These track progress and expose gaps by analyzing conversation logs and producing actionable reports.

Yes, this benchmark automation operates without external dependencies, using internal scripts to handle the full orchestration. It creates test projects, runs skill injections, launches dev servers, and verifies outcomes independently within your environment.