cli-agent-evaluate

You can evaluate a CLI tool against a single failure mode by running its self-contained check command non-interactively with timeouts, scoring the observed behavior against a spec rubric. This captures auditable evidence like exit codes and stdout/stderr for reliable agent testing.

CLI agent spec compliance evaluation targets one documented failure mode per run to pinpoint root causes quickly. It executes a profile-aware, non-interactive check and assigns an evidence-based score from 0 to 3 to prevent AI agents from using unreliable command-line tools.

You can locate failure-mode reference sections for CLI testing using identifiers like section numbers (§N), keywords, or reference file paths. The evaluation targets exactly one failure mode per run to isolate the root cause and generate a reproducible trace artifact.

When a CLI tool scores below 3 on a failure mode check, the evaluation returns an applicable agent workaround. It also persists compact findings and a reproducible trace under the evaluations directory to document the unreliable behavior.

Yes, you must load or create an environment profile to ensure consistent, non-interactive invocation of the CLI tool. This profile-aware execution guarantees the check command runs reliably with timeouts to capture accurate exit codes and output streams.

CLI testing times out during non-interactive execution when the check command exceeds the configured timeout limit. This results in a score of ?/3, and the system persists the trace artifacts, including commands and output, to help diagnose the hanging behavior.