skills.rest
haabe
haabeCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

prompt-optimizer

Compare CLAUDE.md instruction variants against baselines using evaluation reports.

Updated Apr 7, 2026
Search Tags:#workflow#claude#evaluation#benchmark#prompt#experiments#ab-testing
One-click install
npx skills add https://github.com/haabe/tic-tac-toe --skill prompt-optimizer-haabe
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: prompt-optimizer
Source: https://github.com/haabe/tic-tac-toe/tree/main/.claude/skills/prompt-optimizer
Command: npx skills add https://github.com/haabe/tic-tac-toe --skill prompt-optimizer-haabe

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Systematically improve Mycelium instructions through measurement and iteration by running evals, capturing baselines, and comparing variants.

Core Features & Use Cases

  • Baseline capture: record current CLAUDE.md performance to a baseline metric file.
  • Variant testing: apply CLAUDE.md changes, run evaluations, and store per-variant results.
  • Report generation: compare baselines and variants to guide next steps.
  • Exemplar capture: preserve winning instruction trajectories after clean evaluations.
  • End-to-end workflow: support hypothesis, testing, comparison, and decision-making.

Quick Start

Start by capturing the baseline performance with the eval runner, then create and test a CLAUDE.md variant against the baseline.

Frequently Asked Questions about prompt-optimizer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run A/B tests on CLAUDE.md prompts against evaluation benchmarks?▼

To run A/B tests on CLAUDE.md prompts, capture a baseline performance file, apply instruction variants, run evaluations via /eval-runner, and compare the resulting metrics to guide decisions.

How does A/B testing prompt instructions against an evaluation benchmark work?▼

A/B testing prompt instructions works by recording a baseline metric file, applying changes to CLAUDE.md, running evaluations, and storing per-variant results to compare performance against the baseline.

Do I need a specific directory structure to benchmark and compare prompt variants?▼

Yes, you need a versioned storage layout under .claude/optimization to store baseline.json, variant results in results/*.json, and winning instruction trajectories in exemplars/.

What is the best way to systematically improve CLAUDE.md instructions?▼

The best way to improve CLAUDE.md instructions is through data-driven iteration, using an end-to-end workflow that supports hypothesis creation, variant testing, comparison reports, and exemplar capture.

Can I preserve winning prompt trajectories after running evaluation benchmarks?▼

Yes, you can preserve winning instruction trajectories by capturing exemplars after clean evaluations, allowing you to store and reference successful prompt variants for future iterations.

Why does my prompt variant perform worse than the baseline in evaluation reports?▼

Your variant performs worse because evaluation reports compare baseline and variant metrics directly; if the instruction changes degrade task performance, the results/*.json file will reflect lower scores.

Related Skills

Explore More Data & Analytics Skills→
samymitysamymity
1

eval-harness

Implement eval-driven development workflows for Claude Code with pass@k tracking.

Community
Intermediate
AmitabhainArunachalaAmitabhainArunachala
1

Prompt Cybernetics v2.0

Optimize prompt structures by learning from execution feedback and adapting recursion depth.

Community
Advanced
krzemienskikrzemienski
2

performance-regression-detection

Detect performance regressions against baseline metrics above a 10% threshold.

Community
Advanced
Dianshu-LiaoDianshu-Liao
2

llm-evaluation

Implement LLM evaluation with automated metrics and LLM-as-judge patterns.

Community
Advanced