skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using HuggingFace alignment-handbook configs.

Updated Jun 5, 2026
Search Tags:#fine-tuning#huggingface#preference optimization#simpo#deepspeed#llm alignment#dpo alternative
One-click install
npx skills add https://github.com/xu1713/openhorse --skill simpo-training-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/optional-skills/mlops/simpo
Command: npx skills add https://github.com/xu1713/openhorse --skill simpo-training-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a separate reference model, extra GPU memory, and complex configuration. SimPO removes the reference model entirely while outperforming DPO, making preference alignment simpler and more compute-efficient. ## Core Features & Use Cases - Reference-Free Preference Training: Run SimPO training on chosen/rejected preference pairs with beta, gamma-beta ratio, and optional SFT regularization via YAML configs. - Ready-Made Workflows: Prebuilt configurations for Mistral 7B base, Llama 3 8B instruct, and reasoning-intensive math/code tasks with tuned learning rates. - In-Depth References: Detailed guides on loss functions (sigmoid vs hinge), hyperparameter selection by model size, and preference dataset preparation. - Use Case: Fine-tune Llama 3 8B Instruct on the UltraFeedback preference dataset with a single accelerate launch command and DeepSpeed ZeRO-3, without provisioning a reference model. ## Quick Start Train Mistral 7B on the UltraFeedback preference dataset using SimPO with the provided YAML config and DeepSpeed ZeRO-3 accelerate launch.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?▼

Install the HuggingFace alignment-handbook, then launch training with accelerate using a SimPO YAML config that specifies the model, preference dataset, beta, gamma_beta_ratio, and learning rate. The provided Mistral 7B and Llama 3 8B configs work out of the box with DeepSpeed ZeRO-3.

SimPO vs DPO: which preference optimization method should I use?▼

SimPO is reference-free, requiring no reference model, and outperforms DPO by 6.4 points on AlpacaEval 2.0 while being more compute-efficient. Choose DPO when you need a conservative reference-model baseline, and PPO or OpenRLHF for multi-node RL setups.

What learning rate and beta should I use for SimPO training?▼

For 7B models use a learning rate of 3e-7 to 5e-7 with beta between 2.0 and 5.0 and gamma_beta_ratio of 0.5. Use lower learning rates around 3e-7 for math and code reasoning tasks, and reduce both values if the loss diverges.

What dataset format does SimPO training require?▼

SimPO requires preference pairs with prompt, chosen, and rejected fields, such as HuggingFaceH4/ultrafeedback_binarized or argilla/ultrafeedback-binarized-preferences-cleaned. Multiple datasets can be mixed with weights in the dataset_mixer config section.

Why does SimPO training cause the model to forget capabilities?▼

Pure preference optimization can overwrite instruction-following abilities learned earlier. Add SFT regularization by setting sft_weight to 0.1 for instruct models, which mixes a supervised loss on chosen responses into the total loss.

What GPU hardware is needed for SimPO training on 7B models?▼

A 7B model trains on a single A100 40GB GPU using DeepSpeed ZeRO-3 with BF16, gradient checkpointing, and Flash Attention 2. Larger 70B models require around 8 A100 80GB GPUs.

Related Skills

Explore More Software Engineering Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
gqf2008gqf2008

evaluating-llms-harness

Automate language model evaluation across 60+ benchmarks using lm-evaluation-harness.

Community
Advanced
hung-phanhung-phan
1

dataset-curation

Curate AI training datasets for quality, coverage, and quantity.

Community
Advanced
adm-humanerdadm-humanerd

fine-tuning-with-trl

Automate end-to-end RLHF fine-tuning of LLMs with TRL.

Community
Advanced