scaffolding-llm-eval-harness

What problem does it solve?

This Skill turns an empty directory into a reproducible LLM evaluation harness, preventing ad hoc scripts, drifting prompts, and incomparable benchmark runs.

Core Features & Use Cases

• Structured harness scaffolding: Creates the core project files for running multi-scenario evaluations, including runner logic, prompt versioning, scenario storage, and comparison tooling.
• Comparable result tracking: Enforces revision-pinned model IDs, canonical dataset hashing, optional judge-model grading, and JSONL result rows so outputs can be compared across runs.
• Practical evaluation workflows: Fits jailbreak testing, prompt comparisons, fine-tune deltas, summarization checks, and other fixed-test-set LLM evaluation pipelines.
• Guardrails and validation: Refuses to overbuild for one-shot API calls and redirects non-LLM classifier or retrieval evaluations to more appropriate workflows.

Quick Start

Ask this skill to scaffold a versioned LLM evaluation harness for your scenario set, candidate models, and optional judge model.