skills.rest
NousResearch
NousResearchOFFICIAL
Explore all skills by this author →
Website𝕏@nousresearchGitHub

simpo

Train language models with reference-free preference optimization using SimPO.

239k|48.8k|Updated Jul 22, 2025
Search Tags:#preference optimization#rlhf#dpo alternative#reference-free training#LLM alignment#SimPO#Post-Training
One-click install
npx skills add https://github.com/NousResearch/hermes-agent --skill simpo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo
Source: https://github.com/NousResearch/hermes-agent/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/NousResearch/hermes-agent --skill simpo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve?

Preference alignment of large language models traditionally requires a reference model (as in DPO), doubling memory costs and complicating training pipelines. SimPO eliminates this requirement while delivering stronger alignment performance, but practitioners struggle with hyperparameter selection, dataset preparation, and loss function tuning.

Core Features & Use Cases

  • Reference-Free Training: Optimize models directly on preference pairs without loading a reference model, reducing VRAM requirements by half.
  • Hyperparameter Guidance: Select appropriate learning rates, beta values, gamma-beta ratios, and SFT weights for different model sizes and task types.
  • Dataset Preparation: Format, filter, and mix preference datasets like UltraFeedback, HelpSteer, and HH-RLHF for SimPO training.
  • Use Case: Fine-tune Llama 3 8B Instruct on math reasoning preferences using DeepSpeed ZeRO-3 on a single A100 node, achieving better alignment than DPO with lower memory footprint.

Quick Start

Configure and launch SimPO training for Mistral 7B on UltraFeedback preferences using DeepSpeed ZeRO-3 acceleration.

Frequently Asked Questions about simpo

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?▼

Configure a YAML file with model path, dataset mixer, and SimPO hyperparameters (beta, gamma_beta_ratio, learning_rate), then launch with accelerate using a DeepSpeed config. The alignment-handbook provides the run_simpo.py training script.

What is the difference between SimPO and DPO for LLM alignment?▼

SimPO is reference-free, eliminating the need to load a separate reference model and halving memory usage. It uses a target margin (gamma_beta_ratio) and typically requires higher beta values (2.0-10.0) compared to DPO's 0.01-0.1 range.

Does SimPO work with Llama 3 and Mistral instruction models?▼

Yes, SimPO supports both base and instruct variants of Llama 3 8B, Mistral 7B, and Gemma 2 9B. For instruct models, adding sft_weight of 0.05-0.1 helps preserve instruction-following capabilities during alignment.

Why does SimPO training loss diverge during optimization?▼

Loss divergence typically results from learning rate being too high (above 1e-6 for 7B models) or beta being too aggressive. Reduce learning_rate to 3e-7 and beta to 1.0-2.0 to stabilize training.

When should I use SimPO instead of DPO or PPO?▼

Use SimPO when you want simpler training without a reference model, have limited GPU memory, or need better performance than DPO. Choose PPO for maximum control with a reward model, or GRPO for memory-efficient RL without a critic.

Related Skills

Explore More Software Engineering Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
agentic-inagentic-in
577

TRL Fine Tuning

Fine-tune large language models with TRL using SFT, DPO, and PPO/GRPO workflows.

Official
Advanced
matlee0409matlee0409

simpo-training

Optimize language model alignment by directly optimizing log probability differences.

Community
Intermediate
AevonixAevonix

fine-tuning-with-trl

Fine-tune language models with TRL for SFT, DPO, and PPO/GRPO alignment.

Official
Intermediate