run-test-plan

Automating PROBE test runs involves loading a Helm test plan and executing a defined protocol—announce, read, snapshot, execute, evaluate, check side effects, and cleanup—to verify agent behavior. It captures streams, records violations, and updates the scorecard.

Generating a pass/fail scorecard for multiple agents requires orchestrating category-wide test sweeps. The automation loads test plans from artifacts, executes the test protocol across targeted agents, records any violations, and compiles the results into a structured final report.

Test orchestration captures violations during smoke tests by separately recording stdout and stderr streams throughout the execution protocol. It evaluates these captured streams against the test plan, logs any deviations as violations, and automatically updates the scorecard.

Yes, you can run targeted TC tests across multiple agents without manual intervention. The orchestration process loads the specified test plan from artifacts, coordinates the execution protocol across the selected agents, and automatically generates the final pass/fail report.

Helm test orchestration supports category-wide test sweeps across multiple agents. It systematically applies the execution protocol—covering announcement, snapshotting, execution, evaluation, side-effect checking, and cleanup—to verify agent behavior comprehensively and update the scorecard.