prompt-eval

What problem does it solve?

Prompt-eval makes it possible to objectively test whether an AI system prompt triggers the correct tool calls with the correct parameters, without real side effects.

Core Features & Use Cases

• Mock tool-call execution: Runs evaluation runs using a skill-packaged prompt plus a mock tool caller that logs tool invocations to a JSONL file.
• Deterministic grading via assertions: Automatically checks expectations against tool-call logs (and optionally response text) using keyword-based matching.
• Inline and model-comparison modes: Supports running evals in the current session (no extra API cost) or spawning subagent runs to compare models like Haiku vs Sonnet.
• Use Case: Ensure a router or agent prompt correctly calls tools (e.g., task creation, reminders, or clarifications) across a set of behavioral test cases like happy paths, unknown inputs, and multi-step flows.

Quick Start

Ask the AI to package your target system prompt as a skill using a SKILL.md plus a scripts/call_tool.sh mock tool logger, then run the evals defined in evals/evals.json and grade them with grade_evals.py.