skill-tuning-loop

Validates proposed skill edits against mined session evidence and held-out eval cases.

3|Updated Jun 15, 2026
One-click install
npx skills add https://github.com/patrickserrano/lacquer --skill skill-tuning-loop-patrickserrano
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: skill-tuning-loop
Source: https://github.com/patrickserrano/lacquer/tree/main/core/skills/skill-tuning-loop
Command: npx skills add https://github.com/patrickserrano/lacquer --skill skill-tuning-loop-patrickserrano

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Deciding whether a shared skill actually needs an edit is usually a judgment call based on anecdote. This Skill replaces guesswork with an empirical loop: it mines real session transcripts for recurring friction, proposes a bounded fix, and validates it against held-out cases before anything ships. ## Core Features & Use Cases - Friction Mining: Searches recorded sessions via agentsview MCP tools for skill invocations plus friction phrases, requiring evidence from multiple independent sessions before acting. - Bounded Proposals: Generates narrow diffs touching only the lines implicated by the evidence, still satisfying the skill-authoring-standard rubric. - Validation Gate: Runs held-out eval cases against old and new skill text in parallel, judged by an independent stronger model; accepts only if nothing regresses and at least one case improves. - Use Case: A skill in core/skills/ keeps getting corrected the same way across unrelated sessions. Run this loop to confirm the pattern, produce an evidence-backed diff, and open a reviewable PR — a human always merges. ## Quick Start Run the skill-tuning-loop workflow on the skill that keeps getting corrected to mine session evidence and validate a bounded fix before opening a PR.

Frequently Asked Questions about skill-tuning-loop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test whether a skill edit actually helps?▼

Run the skill-tuning-loop workflow: mine session transcripts for recurring friction, propose a bounded edit, then validate old versus new skill text against held-out eval cases judged by an independent stronger model. Accept only if nothing regresses and at least one case improves.

When should I run a skill tuning pass?▼

Run it when a skill keeps needing the same correction across unrelated sessions, or when triaging an automation-proposed edit. Never run it on every session or on a single session's evidence — one correction is noise, not a pattern.

What is the difference between skill-tuning-loop and skill-authoring-standard?▼

skill-authoring-standard is a static prose rubric checked manually by a human. skill-tuning-loop is an empirical, evidence-gated loop checked by rollouts against held-out cases — it answers whether a proposed edit measurably helps, not just whether it reads well.

Can the tuning loop merge accepted skill edits automatically?▼

No. An ACCEPT verdict produces a diff with its evidence — session excerpts and before/after eval scores — for a PR that a human merges. The loop never commits, pushes, or merges on its own.

Why does the loop distrust claims that a tool or skill does not exist?▼

Reflect and Propose can only grep the repo's committed files, so system-level tools and plugin-provided skills installed elsewhere are invisible to them. Edits justified by a missing tool or skill are forbidden because that class of mistake reads as confident right up until it is wrong.