skills.rest
wshobson
wshobsonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

eval-harness-first

Build evaluation harnesses with goldens, graders, and baselines that gate fine-tuning runs.

39.3k|4.2k|Updated Jul 24, 2025
Search Tags:#fine-tuning#baseline#LLM evaluation#grader#golden dataset#llm judge#eval harness
One-click install
npx skills add https://github.com/wshobson/agents --skill eval-harness-first
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness-first
Source: https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/eval-harness-first
Command: npx skills add https://github.com/wshobson/agents --skill eval-harness-first

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires jsonschema, pytest, and includes references (resource) components.

What problem does it solve?

Fine-tuning without an evaluation harness leaves nothing to measure against, no way to catch regressions, and no labeled data to train on. This Skill establishes the Phase 0 gate: a versioned eval harness built from error analysis before any training config is written.

Core Features & Use Cases

  • Golden Set Construction: Convert production traces via open/axial coding into 4-8 failure buckets, or generate synthetic goldens by dimension-based sampling when traces don't exist.
  • Per-Bucket Graders: One deterministic-first grader per failure bucket (schema, exact-match, execution-based), with calibrated LLM-judges reserved for genuinely subjective criteria.
  • Judge Calibration Protocol: TPR/TNR reporting on sealed test splits, snapshot pinning, bias correction, and a different-model-family requirement.
  • Baseline Gating: Run the full harness against the base model to produce eval/baseline-<model>.json, the gate token every later checkpoint is compared against.
  • Use Case: Starting an LLM fine-tuning effort, convert 100+ production traces into a versioned eval/ directory with goldens, graders, a drift suite, and a base-model baseline before selecting a training method.

Quick Start

Use the eval-harness-first skill to build an evaluation harness from my production traces before I start fine-tuning.

Frequently Asked Questions about eval-harness-first

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an evaluation harness before fine-tuning an LLM?▼

Run error analysis on at least 100 production traces using open coding, collapse tags into 4-8 failure buckets via axial coding, then write one grader per bucket. Commit the goldens as versioned JSONL and run the harness against the base model to produce a baseline file.

When should I use an LLM judge versus a deterministic grader?▼

Use deterministic graders (regex, schema validation, execution checks) first since they are cheaper and reproducible. Reserve LLM judges for genuinely subjective criteria like tone or faithfulness where no deterministic check can express the criterion.

How do I calibrate an LLM judge against human labels?▼

Label at least 100 items, split into train, dev, and sealed test sets, then report TPR and TNR rather than blended accuracy. Pin the judge to a fixed model snapshot from a different model family than the model under test, and recalibrate quarterly or on judge-model change.

Why should drift-suite scoring use logprobs instead of generate-and-extract?▼

Generate-and-extract is parse-brittle for models that preamble before answering, conflating format compliance with the knowledge being measured. Logprob scoring compares answer-letter token probabilities directly, so verbose models score on their actual answer distribution.

What happens if my LLM judge cannot meet the TPR/TNR bar?▼

The judge ships advisory-only: its verdicts flag items for human review but never gate checkpoint promotion or count into reported pass rates. Prefer reframing the criterion as a deterministic grader over lowering the calibration bar.

Can I skip judge calibration if all my graders are deterministic?▼

Yes. An all-deterministic harness has nothing to calibrate, so the calibration step is N/A. State that explicitly in the Phase 0 exit checklist rather than inventing a subjective criterion just to have something to calibrate.

Related Skills

Explore More Data & Analytics Skills→
iani-kuliiani-kuli

eval-audit

Audit LLM evaluation pipelines and prioritize systemic issues in error analysis and judge design.

Community
Advanced
agentic-inagentic-in
577

Training

Plan fine-tuning or post-training workflows from objectives, data, and hardware constraints.

Official
Advanced
qodex-aiqodex-ai
39

llm-fine-tuning-guide

Guide fine-tuning of large language models with LoRA and QLoRA techniques.

Official
Advanced
doanchienthangdevdoanchienthangdev
4

LLMOps

Manages LLM lifecycle including prompts, RAG pipelines, versioning, A/B testing, evaluation, deployment, monitoring, and cost tracking.

Community
Advanced