prompt-eval

You can test if a system prompt triggers correct tool calls by packaging it as a skill with a mock tool caller that logs invocations to a JSONL file, then running deterministic grading assertions against those logs to verify expected behavior.

Yes, you can compare model performance by spawning subagent runs to evaluate models like Haiku vs Sonnet, using mock tool-call execution to log invocations and grade behavior deterministically without real side effects.

To set up tool-calling prompt evaluations, you need a root SKILL.md with tool-calling instructions, a scripts/call_tool.sh mock tool logger for JSONL output, an evals/evals.json test suite, and a grade_evals.py script to compare expected vs observed logs.

Mock tool-call evaluations handle multi-step and edge-case scenarios by running a test suite of behavioral cases—including happy paths, unknown inputs, and multi-step flows—against the skill-packaged prompt to ensure correct task routing and tool invocation.

Yes, prompt testing for agent workflows supports inline evaluation runs within the current session to avoid extra API costs, as well as subagent runs for broader model comparison across varied user messages.

Deterministic grading assertions are applied by automatically checking expectations against logged tool-call parameters and optionally response text, using keyword-based matching to compare expected vs observed tool-call logs.