skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

simpo

Train language models with reference-free SimPO preference optimization using HuggingFace alignment-handbook.

Updated Sep 10, 2026
Search Tags:#fine-tuning#preference optimization#DPO alternative#simpo#post-training#llm alignment#reference-free
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill simpo-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill simpo-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning language models to human preferences with DPO requires maintaining a separate reference model, doubling memory and compute. SimPO removes the reference model entirely while outperforming DPO, but getting the hyperparameters (beta, gamma-beta ratio, learning rate) right is tricky and mistakes cause loss divergence or capability forgetting. ## Core Features & Use Cases - Reference-free preference training: Launch SimPO training runs on models like Mistral 7B and Llama 3 8B using accelerate and DeepSpeed ZeRO-3 configs. - Hyperparameter guidance: Model-size and task-specific recommendations for beta, gamma-beta ratio, learning rate, and SFT weight, including reasoning-intensive settings for math and code. - Dataset preparation: Preference data formats, quality filtering, deduplication, and mixing strategies for UltraFeedback, HH-RLHF, and custom datasets. - Use Case: Fine-tune Llama 3 8B Instruct on cleaned UltraFeedback preferences with beta 2.5 and a small SFT weight to improve alignment without losing instruction-following ability. ## Quick Start Ask the agent to set up a SimPO training run for Mistral 7B on the UltraFeedback dataset with the recommended default hyperparameters.

Frequently Asked Questions about simpo

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?▼

Install the HuggingFace alignment-handbook, then launch training with accelerate using a SimPO YAML config that sets the model, dataset_mixer, beta, gamma_beta_ratio, and learning rate. For Mistral 7B, use learning_rate 5e-7, beta 2.0, and DeepSpeed ZeRO-3.

SimPO vs DPO: which preference optimization method should I use?▼

SimPO is simpler and removes the reference model, reducing memory and compute while often outperforming DPO. Choose DPO when you need an established reference-model baseline or more conservative updates; choose PPO or GRPO via OpenRLHF for multi-node RL setups.

What learning rate and beta should I use for SimPO on a 7B model?▼

For 7B-8B models, use a learning rate between 3e-7 and 5e-7, with 3e-7 for math and code reasoning tasks. Set beta between 2.0 and 5.0 with gamma_beta_ratio around 0.5; higher beta and ratio suit strong, clear preference data.

Why does SimPO training loss diverge and how do I fix it?▼

Loss divergence usually means the learning rate or beta is too high. Reduce learning_rate from 5e-7 to 3e-7 and lower beta from 2.0 to 1.0, then monitor the reward margin between chosen and rejected responses.

How do I prevent capability forgetting during SimPO fine-tuning?▼

Add SFT regularization by setting sft_weight between 0.05 and 0.2, which mixes cross-entropy loss on chosen responses into the SimPO loss. This is recommended when fine-tuning instruct or chat models rather than base models.

What GPUs are required for SimPO training on 7B to 70B models?▼

A 7B model fits on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model needs eight A100 80GB. Enable BF16, gradient checkpointing, and Flash Attention 2 to reduce memory.

Related Skills

Explore More Software Engineering Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
MLT-OSSMLT-OSS
14

evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Official
Advanced
SethyPagnaSethyPagna

fine-tuning-with-trl

Fine-tune language models with TRL for post-training alignment workflows.

Community
Advanced
OpenLAIROpenLAIR
1.0k

openrlhf-training

Run distributed RLHF training for large language models on Ray clusters.

Official
Advanced