skills.rest
selcukyucel
selcukyucelCOMMUNITY
Explore all skills by this author →
GitHub

prompt-test

Run a prompt against test inputs and score outputs with a binary rubric.

1|Updated Mar 26, 2026
Search Tags:#evaluation#reporting#baseline#rubric#prompt-testing#ai-prompts
One-click install
npx skills add https://github.com/selcukyucel/north-starr-genai --skill prompt-test-selcukyucel
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: prompt-test
Source: https://github.com/selcukyucel/north-starr-genai/tree/main/skills/prompt-test
Command: npx skills add https://github.com/selcukyucel/north-starr-genai --skill prompt-test-selcukyucel

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Run a single-shot evaluation of a prompt against a test suite of inputs, score outputs using a binary rubric, and report a pass/fail verdict. It serves as the one-shot counterpart to /autoimprove, letting you gauge impact after a manual prompt change without iterative loops.

Core Features & Use Cases

  • Locate inputs from the user's prompt and test suite references
  • Load the evaluation rubric and an optional baseline for comparison
  • Execute the prompt against all test inputs, capture and score outputs
  • Repeat runs for non-deterministic outputs and aggregate per-input results
  • Save detailed results under .plans/EVAL-<name>/ and optionally set a new baseline

Quick Start

Provide the prompt text and a test suite, then run the evaluation to produce a results report.

Frequently Asked Questions about prompt-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an AI prompt against a test suite?▼

To evaluate an AI prompt, prompt-test runs the prompt against a suite of test inputs and scores the outputs using a binary rubric. It then aggregates the results and generates a structured pass/fail report saved under .plans/EVAL-<name>/.

What is single-run prompt evaluation with a binary rubric?▼

Single-run prompt evaluation is a one-shot process that scores prompt outputs against a binary rubric to determine pass or fail. It serves as a counterpart to iterative auto-improvement, letting you gauge the impact of manual prompt changes quickly.

How do I handle non-deterministic prompt outputs during testing?▼

To handle non-deterministic prompt outputs, prompt-testing supports multiple runs of the same test inputs. It aggregates the per-input results across these runs to determine a reliable pass/fail verdict against the evaluation rubric.

Can I compare prompt evaluation results against a previous baseline?▼

Yes, prompt-test can compare current evaluation results against an optional saved baseline. If the new results are an improvement, you can optionally update the baseline to reflect the new standard for future comparisons.

What do I need to run a prompt evaluation?▼

You need the prompt text you want to test and a test suite of inputs, along with an evaluation rubric. The Skill locates these references, executes the prompt against the inputs, and outputs a structured pass/fail report.

Where are prompt evaluation reports saved?▼

Prompt evaluation reports are saved under the .plans/EVAL-<name>/ directory. This structured output includes detailed results of the test suite execution and the binary rubric scoring.

Related Skills

Explore More Software Engineering Skills→
selcukyucelselcukyucel
1

eval-suite

Generate structured evaluation datasets with binary rubric scoring for AI outputs.

Community
Advanced
openJiuwen-aiopenJiuwen-ai
2.5k

skill-creator

Draft SKILL.md content, generate test prompts, and run trigger-evaluation benchmarks.

Official
Advanced
akillnessakillness
40

skill-autoresearch

Optimize existing skills by scoring test outputs with binary evals.

Community
Advanced
liuerfireliuerfire
10

prompt-engineer

Designs, optimizes and evaluates prompts for LLMs using techniques like chain-of-thought and few-shot learning.

Community
Advanced