skills.rest
DevOtts
DevOttsCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

make-eval

Build deterministic evaluation harnesses for LLM classifiers with exact-match scoring.

4|Updated Jun 27, 2025
Search Tags:#regression testing#guardrails#LangSmith#confusion matrix#llm evals#classifier testing#exact match
One-click install
npx skills add https://github.com/DevOtts/all-skills-you-will-ever-need --skill make-eval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: make-eval
Source: https://github.com/DevOtts/all-skills-you-will-ever-need/tree/main/.claude/make-eval
Command: npx skills add https://github.com/DevOtts/all-skills-you-will-ever-need --skill make-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langsmith, and includes scripts (resource) components.

What problem does it solve?

LLM-backed classifiers can silently regress, misroute, or fail guardrails, and teams often only discover that after users are impacted. This Skill builds a small, deterministic evaluation harness that measures exact-match accuracy and pinpoints dangerous misclassifications with an auditable confusion matrix.

Core Features & Use Cases

  • Closed-label exact-match scoring: scores predictions with strict got === expected for safer, more deterministic validation.
  • Adversarial dataset guidance: helps you create a compact dataset covering happy paths, empty/garbage inputs, prompt injection, ambiguous cases, contradictions, non-English, and the single most dangerous failure mode.
  • CI-friendly regression gate: prints a confusion matrix and exits non-zero when accuracy falls below a threshold or any DANGEROUS_* category is missed.
  • Optional LangSmith mode: when LangSmith is wired, it generates upload/run scripts, wires pnpm eval workflows, and reports experiment + dataset links on every run.

Quick Start

Ask an AI agent to generate the eval harness for your classifier by providing your closed label set and the boundary function that returns the predicted label, then run it with your dataset to get an exact-match confusion matrix.

Frequently Asked Questions about make-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run regression testing for an LLM classifier without using LLM-as-judge?▼

Regression testing for an LLM classifier requires a deterministic evaluation harness that scores exact-match accuracy, verifying that predicted labels strictly match expected labels to ensure safe, repeatable validation.

What's the best way to catch silent misclassifications in LLM intent routing?▼

Catching silent misclassifications in LLM intent routing requires an auditable confusion matrix built from adversarial input datasets, pinpointing exact-match failures and flagging critical routing errors before users are impacted.

Does LangSmith support exact-match evaluation harnesses for CI gating?▼

LangSmith supports exact-match evaluation harnesses by generating upload and run scripts, wiring pnpm eval workflows, and reporting experiment and dataset links on every run when integrated into your CI gating pipeline.

How do I build a regression gate that stops CI on dangerous LLM guardrail misses?▼

Building a regression gate that stops CI on dangerous LLM guardrail misses requires an evaluation harness that exits non-zero when accuracy falls below a threshold or any DANGEROUS_* category is missed during exact-match scoring.

What inputs do I need to test closed-label LLM tagging accuracy?▼

Testing closed-label LLM tagging accuracy requires auditable input-to-expected datasets covering happy paths, empty inputs, prompt injection, ambiguous cases, and contradictions, paired with your classifier's boundary prediction function.

Can I use exact match scoring for priority tagging and yes/no extraction workflows?▼

Exact match scoring is ideal for priority tagging and yes/no extraction workflows because it applies strict predicted-versus-expected label comparison, ensuring deterministic validation for closed label sets.

Related Skills

Explore More Software Engineering Skills→
matlabmatlab
162

matlab-test-generator

Generate MATLAB unit tests using the MATLAB Testing Framework.

Official
Intermediate
wpsadiwpsadi

langsmith-dataset

Manage LangSmith datasets via CLI for AI model evaluation.

Community
Advanced
raphaelchristiraphaelchristi
43

harness:health

Detect and auto-correct quality issues in LangSmith evaluation datasets.

Community
Advanced
doanchienthangdevdoanchienthangdev
4

evaluation-methodology

Evaluate AI model outputs with exact match, semantic similarity, LLM-as-judge, and ELO ranking.

Community
Intermediate