skills.rest
wshobson
wshobsonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

checkpoint-promotion

Gate fine-tuned checkpoints through drift budgets, paired arenas, and canary rollouts before promotion.

39.3k|4.2k|Updated Jul 24, 2025
Search Tags:#drift detection#evaluation#fine-tuning#checkpoint#lLM#model promotion#catastrophic forgetting
One-click install
npx skills add https://github.com/wshobson/agents --skill checkpoint-promotion
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: checkpoint-promotion
Source: https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/checkpoint-promotion
Command: npx skills add https://github.com/wshobson/agents --skill checkpoint-promotion

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Fine-tuned checkpoints that beat their task metric can still silently degrade general capability, leak eval goldens into training data, or lose live comparisons against the base model. This Skill enforces a four-stage promotion gate so only checkpoints that clear data-quality, drift, arena, and canary checks ship.

Core Features & Use Cases

  • Four-Stage Promotion Gate: Sequentially gates checkpoints through data-quality checks (dedup, goldens leakage, label noise), a frozen capability-drift suite diffed against a baseline, a position-randomized paired arena versus the base model, and a 5-10% canary rollout with auto-rollback.
  • Drift Budget Enforcement: Applies a strict verdict table where drift over 5 percentage points is a hard fail regardless of task-metric gains, with CI half-width reporting and seed-variation reruns for the 2-5pt band.
  • Catastrophic Forgetting Remediation: Provides an ordered escalation ladder (replay-mix fraction, learning rate, epochs, LoRA rank) and produces a terminal PROMOTE or REJECT verdict with exactly one top remediation in a promotion-report.md.
  • Use Case: After a LoRA fine-tuning run finishes, run this gate to re-run the frozen drift suite against the baseline, compare the checkpoint against the base model in a paired arena, and emit a promotion report that downstream commands consume directly.

Quick Start

Evaluate my newly trained checkpoint against the baseline eval results and produce a promotion report with a PROMOTE or REJECT verdict.

Frequently Asked Questions about checkpoint-promotion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I decide whether a fine-tuned checkpoint is safe to deploy?▼

Run a four-stage promotion gate: check training data quality, re-run a frozen capability-drift suite against the baseline, compare the checkpoint against the base model in a paired arena, and canary at 5-10% traffic. The gate ends in a terminal PROMOTE or REJECT verdict with evidence.

What is a drift budget in LLM fine-tuning evaluation?▼

A drift budget defines acceptable capability loss in percentage points: 1pt or less is noise, 2-5pt requires a seed-variation rerun, and over 5pt is a hard fail. Task-metric gains never override a drift-budget breach on general benchmarks like MMLU or GSM8K.

How do I fix catastrophic forgetting after LoRA fine-tuning?▼

Follow an ordered escalation ladder: first swap the replay-mix fraction toward more general-domain data while holding step count constant, then lower the learning rate, then reduce epochs, then shrink the LoRA rank. Treat single-run lever attribution as a hypothesis until a seed repeat confirms it.

Can I skip the canary stage for a local-only model deployment?▼

Yes. Local-only deployments correctly stop after stage 3, the paired arena comparison. The 5-10% stratified canary with auto-rollback applies only to checkpoints receiving production traffic.

Why does a checkpoint that wins on goldens still get rejected?▼

A holdout win that loses the live paired arena does not ship. Stage 2 drift numbers and stage 3 arena judgments must agree; a frozen-goldens win combined with a paired-comparison loss is treated as a real failure signal, not a discrepancy to explain away.

How many eval items do I need for a reliable drift verdict?▼

The strict sample size for a 2.5pt half-width at typical accuracy is about 1,300 items; 200 is a pragmatic floor with roughly a 6pt half-width. Always report the CI half-width, and treat any margin smaller than it as REJECT (uncertain) rather than pass or fail.

Related Skills

Explore More Data & Analytics Skills→
BedaBeda-GrowthBedaBeda-Growth

run-qa-loop

Automate the QA loop against a frozen golden-master to detect drift and require approval to refreeze.

Official
Advanced
cuba6112cuba6112

unsloth-gguf

Export fine-tuned machine learning models to GGUF format with quantization and LoRA merging.

Community
Intermediate
AlsKozlovAlsKozlov
17

training-data-review

Audit AI training datasets for Russian legal compliance.

Community
Intermediate
thinking-machines-labthinking-machines-lab
4.0k

rlhf

Orchestrates an RLHF workflow with SFT, RM training, and RL via the Tinker API.

Official
Advanced