skills.rest
wshobson
wshobsonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

preference-optimization

Align fine-tuned language models with preference data using DPO, ORPO, KTO, or SimPO.

39.3k|4.2k|Updated Jul 24, 2025
Search Tags:#preference optimization#trl#DPO#LLM alignment#SimPO#KTO#ORPO
One-click install
npx skills add https://github.com/wshobson/agents --skill preference-optimization
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: preference-optimization
Source: https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/preference-optimization
Command: npx skills add https://github.com/wshobson/agents --skill preference-optimization

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trl, unsloth, and includes references (resource) components.

What problem does it solve?

Choosing and configuring a preference-optimization method for LLM alignment is error-prone: teams force unpaired feedback into DPO, port SFT learning rates into DPO runs, or chase DPO variants that don't hold up at scale. This Skill routes preference data to the right method and produces validated TRL configs.

Core Features & Use Cases

  • Method Selection: Routes preference pairs to DPO, memory-bound or SFT-free setups to ORPO, unpaired thumbs-up/down feedback to KTO, and length-bias cases with sweep budget to SimPO.
  • Production Training Pattern: Describes iterative on-policy DPO where each round's checkpoint becomes the next round's reference model.
  • Pair Construction: Builds chosen/rejected pairs from same-task passing-vs-failing trajectories using μ−2σ rejected selection instead of naive best-vs-worst.
  • Use Case: A team with an SFT checkpoint and reviewer thumbs-up/down feedback gets routed to KTO with a complete KTOConfig block, avoiding the mistake of synthesizing fake preference pairs for DPO.

Quick Start

Ask the assistant to choose and configure a preference-optimization method for your SFT checkpoint given your feedback data format.

Frequently Asked Questions about preference-optimization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between DPO, ORPO, KTO, and SimPO for preference tuning?▼

Use DPO as the default for paired preference data with β=0.1 and LR 5e-7 to 1e-6. Choose ORPO when memory-bound or lacking an SFT checkpoint, KTO for unpaired thumbs-up/down feedback, and SimPO only when length bias appears and sweep budget exists.

What learning rate should I use for DPO training?▼

Use 5e-7 to 1e-6 for 1-2 epochs, which is lower than the SFT learning rate that produced the checkpoint. Porting an SFT-scale LR into DPO is the most common misconfiguration and typically causes catastrophic forgetting.

Can I use DPO with unpaired thumbs-up/down feedback?▼

No, use KTO instead for unpaired binary feedback. KTO takes a prompt, completion, and boolean label per example without requiring matched pairs, so don't synthesize artificial pairs to force DPO onto unpaired data.

DPO vs ORPO: when is ORPO the better choice?▼

ORPO is better when GPU memory is constrained or no separate SFT checkpoint exists. It is reference-free and fuses the SFT and preference objectives into one loss, eliminating DPO's reference-model memory cost.

Why does my preference-tuned model lose general capability?▼

Catastrophic forgetting after preference tuning is almost always caused by a too-high learning rate. Drop the LR toward the low end of the method's range first, then reduce epochs, and only then consider mixing 10-30% general instruction data into the run.

How should I construct chosen and rejected pairs for DPO?▼

Build pairs from same-task passing-versus-failing trajectories, selecting the rejected member at μ−2σ of the reward distribution rather than the absolute minimum. Naive best-vs-worst pairing degrades as model scale increases.

Related Skills

Explore More Software Engineering Skills→
JKhyroJKhyro

fine-tuning-with-trl

Fine-tune language models with TRL for SFT, DPO, PPO, and GRPO training.

Community
Advanced
cabeznocabezno

fine-tuning-with-trl

Fine-tune transformer language models with TRL methods for RLHF-style alignment.

Community
Advanced
MileniumTickMileniumTick

edge-strategy-reviewer

Analyze YAML edge strategy drafts and return PASS, REVISE, or REJECT verdicts.

Community
Advanced
Strategic-AutomationStrategic-Automation
6

fine-tuning-with-trl

Combine supervised fine-tuning and reinforcement learning with TRL for model alignment.

Official
Advanced