cli-agent-evaluate-batch

You can batch test CLI tools for AI agent failure modes by running a comprehensive evaluation across multiple CLI Agent Spec failure modes. The skill executes deterministic invocations and produces a scored batch report with critical, high, and medium severity outcomes for remediation.

A CLI agent evaluation scorecard is a final summary table produced after testing a CLI against various failure modes. It aggregates per-failure-mode outcomes across critical, high, and medium severity tiers to help developers identify and fix observability and regression issues.

Yes, you can resume an interrupted CLI evaluation because the skill manages a resumable queue using findings and trace completeness. It loads prior environment artifacts, skips fully completed checks, and saves progress incrementally after each failure mode evaluation.

Timeouts during AI agent CLI batch testing are recorded as indeterminate scores, marked as ?/3 in the evaluation results. This ensures that unresponsive CLI invocations do not corrupt the final scorecard and are explicitly tracked for further investigation.

Environment discovery is required before evaluating a CLI for AI agents when needed, specifically via the cli-agent-onboard workflow. This setup ensures the skill can perform deterministic invocations per the stored profile during the subsequent batch testing process.

AI agent failure mode testing covers CLIs invoked for deployment, querying, file operations, and pipelines. The evaluation applies across various spec parts and severity tiers to ensure AI agents do not silently hang or misinterpret CLI behavior during these operations.