jarvis-test-runner

Automate multi-trial testing for Jarvis plugins by executing test suites with configurable trials and pass@k metrics. Scoping test data from tests/agents/*.tests.json and tests/skills/*.tests.json validates responses against structured assertions to ensure reliability.

Pass@k metrics in automated QA testing quantify stability across non-deterministic outputs by executing multiple trials. Pass@k, pass^k, and majority metrics measure consistency to surface failures and prevent regressions in plugin development.

Run end-to-end tests for Jarvis agents and skills by discovering structured test data from tests/agents/*.tests.json and tests/skills/*.tests.json. The runner validates responses against assertions and generates detailed human-readable and machine-friendly reports.

Yes, structured assertions are required to validate Jarvis plugin responses. The automated test runner scopes data from JSON test files and checks responses against these assertions to produce detailed reports and prevent regressions.

Yes, you can generate machine-friendly and human-readable reports for Jarvis plugin test failures. The automated test workflow executes configured trials, validates responses against structured assertions, and outputs detailed reports to surface failures.