skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using HuggingFace alignment-handbook configs.

14|5|Updated Apr 9, 2026
Search Tags:#fine-tuning#huggingface#preference optimization#simpo#deepspeed#llm alignment#dpo alternative
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill simpo-training-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill simpo-training-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs to human preferences with DPO or PPO requires a reference model or reward model, adding memory overhead and training complexity. This Skill guides SimPO training, a reference-free preference optimization method that outperforms DPO (+6.4 points on AlpacaEval 2.0) with simpler, more efficient single-node training. ## Core Features & Use Cases - Reference-Free Preference Training: Train on chosen/rejected preference pairs without loading a reference model, using the HuggingFace alignment-handbook with DeepSpeed ZeRO-3. - Hyperparameter Guidance: Provides recommended beta, gamma_beta_ratio, learning rate, and sft_weight values per model size (7B to 70B) and task type (chat, math, code). - Dataset & Troubleshooting References: Covers preference dataset formats, quality filtering, loss function selection (sigmoid vs hinge), and fixes for loss divergence, capability forgetting, and OOM. - Use Case: Fine-tune Mistral 7B on UltraFeedback preference data with a single accelerate launch command, or align Llama 3 8B Instruct on math preferences with SFT regularization to preserve capabilities. ## Quick Start Train Mistral 7B with SimPO on the UltraFeedback dataset using the provided YAML config and accelerate launch with DeepSpeed ZeRO-3.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO instead of DPO?▼

Install the HuggingFace alignment-handbook, prepare a preference dataset with prompt/chosen/rejected fields, and launch scripts/run_simpo.py with a YAML config via accelerate. SimPO needs no reference model, so training is simpler and more memory-efficient than DPO.

SimPO vs DPO vs PPO: which preference optimization method should I use?▼

Use SimPO for the simplest setup with strong performance and no reference model. Choose DPO when you need a conservative reference-model baseline, PPO when you need maximum control with a reward model, and OpenRLHF for multi-node distributed RL training.

What learning rate and beta should I use for SimPO training?▼

For 7B models use learning rate 3e-7 to 5e-7 with beta 2.0-2.5 and gamma_beta_ratio 0.5. Use lower learning rates (around 3e-7) for math and code reasoning tasks, and higher beta (5.0) for stronger preference signals.

What GPU hardware is required for SimPO training?▼

A 7B model trains on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model needs eight A100 80GB. Enable BF16, gradient checkpointing, and Flash Attention 2 to reduce memory usage.

Why does SimPO training loss diverge or the model forget capabilities?▼

Loss divergence usually means the learning rate or beta is too high; reduce learning rate to 3e-7 and beta to 1.0. For capability forgetting on instruct models, add SFT regularization with sft_weight between 0.05 and 0.2.

What dataset format does SimPO preference training require?▼

Datasets need prompt, chosen, and rejected fields containing the instruction and the preferred and dispreferred responses. Popular ready-to-use options include HuggingFaceH4/ultrafeedback_binarized, argilla/ultrafeedback-binarized-preferences-cleaned, and Anthropic/hh-rlhf.

Related Skills

Explore More Data & Analytics Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
thinking-machines-labthinking-machines-lab
4.0k

sft

Configure and execute supervised fine-tuning of LLMs using the Tinker API.

Official
Advanced
eddieluengeddielueng
1

fine-tuning-with-trl

Fine-tune language models with TRL for SFT, DPO, and PPO/GRPO workflows.

Community
Advanced