skills.rest
langwatch
langwatchOFFICIAL
Explore all skills by this author →
GitHub

test-cli-usability

Write scenario tests verifying CLI tools work non-interactively for AI agents.

3.5k|362|Updated Sep 9, 2023
Search Tags:#CLI#usability testing#Vitest#scenario testing#non-interactive#AI Agents#langwatch
One-click install
npx skills add https://github.com/langwatch/langwatch --skill test-cli-usability
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-cli-usability
Source: https://github.com/langwatch/langwatch/tree/main/skills/_compiled/native/test-cli-usability
Command: npx skills add https://github.com/langwatch/langwatch --skill test-cli-usability

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires @langwatch/scenario, vitest, @ai-sdk/openai.

What problem does it solve?

CLI tools often hang on interactive prompts, produce unclear errors, or lack discoverable help, causing AI coding agents like Claude Code to get stuck or resort to fragile workarounds like piping 'yes' into commands.

Core Features & Use Cases

  • Non-interactivity verification: Scenario tests assert that agents never use workarounds like yes |, expect -c, or piping newlines into stdin.
  • Error recovery testing: Scenarios where the agent makes mistakes (wrong command, missing argument, auth failure) and must self-correct using --help and error output.
  • Judge-based evaluation: Uses @langwatch/scenario with a user simulator and judge agent to score whether the CLI was used correctly.
  • Use Case: You maintain a deployment CLI and want to prove Claude Code can run every command without hanging on confirmation prompts before shipping it to users.

Quick Start

Write a scenario test that has an AI agent discover and run my CLI's commands, and flag any command that requires interactive input.

Frequently Asked Questions about test-cli-usability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test if my CLI works with AI agents?▼

Write scenario tests with @langwatch/scenario where an agent adapter attempts real tasks using your CLI, a user simulator provides goals, and a judge agent scores whether commands were used correctly without interactive prompts.

How to make a CLI tool agent-friendly?▼

Provide non-interactive flags like --yes, --force, or --non-interactive for every prompt, ensure --help works on every subcommand, use non-zero exit codes for failures, and write error messages with actionable guidance.

What dependencies does @langwatch/scenario require?▼

Install @langwatch/scenario, vitest, and @ai-sdk/openai via npm for TypeScript tests. A Python alternative is available via pip install langwatch-scenario with pytest.

Why does my CLI hang when an AI agent runs it?▼

The CLI likely waits on stdin for confirmation or input that never arrives. Add flag-based alternatives to every interactive prompt so agents can pass answers as arguments instead of typing into a prompt.

Can I test CLI error recovery with scenario tests?▼

Yes. Write scenarios where the agent uses a wrong command name, omits a required argument, or hits an authentication failure, then judge whether it reads --help or error output and self-corrects.

Related Skills

Explore More Software Engineering Skills→
semaphoreuisemaphoreui
14.1k

semaphore-cli-docs

Generate and synchronize CLI documentation pages from Cobra command code.

Official
Advanced
jtvarisjtvaris

tdd-bug-fix

Reproduce bugs with a failing test, then implement a minimal fix.

Community
Intermediate
victorsteinvictorstein

agent-monitor

Monitor Claude Code sub-agent activity with a TypeScript CLI.

Community
Intermediate
SeanChenRSeanChenR

adk-eval-guide

Diagnose ADK evaluation failures and recommend fixes for agent results.

Community
Advanced