skills.rest
vivekgoquest
vivekgoquestCOMMUNITY
Explore all skills by this author →
GitHub

simpo

Train language models with reference-free SimPO preference optimization using TRL and DeepSpeed.

Updated Aug 22, 2026
Search Tags:#fine-tuning#DeepSpeed#preference optimization#DPO alternative#simpo#post-training#llm alignment
One-click install
npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill simpo-vivekgoquest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo
Source: https://github.com/vivekgoquest/hermes-agent-stable/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill simpo-vivekgoquest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning language models to human preferences with DPO requires maintaining a separate reference model, doubling memory usage and complicating training pipelines. SimPO removes the reference model entirely while outperforming DPO, but getting the beta, gamma, and learning rate settings right is non-trivial. ## Core Features & Use Cases - Reference-Free Preference Training: Launch SimPO training runs on preference datasets (UltraFeedback, HH-RLHF, math preferences) using accelerate and DeepSpeed ZeRO-3 configs. - Hyperparameter Guidance: Model-size-specific recommendations for learning rate, beta, gamma-beta ratio, and SFT weight, plus troubleshooting for loss divergence, capability forgetting, and OOM. - Dataset Preparation: Preference data formats, quality filtering, deduplication, and synthetic pair generation workflows. - Use Case: Fine-tune Llama 3 8B Instruct on cleaned UltraFeedback preferences with SFT regularization to preserve instruction-following while improving alignment, all on a single node with two A100 GPUs. ## Quick Start Train Mistral 7B on the UltraFeedback preference dataset using SimPO with the recommended default hyperparameters.

Frequently Asked Questions about simpo

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?▼

Install the alignment-handbook and Flash Attention 2, then launch training with accelerate using a DeepSpeed ZeRO-3 config and a SimPO YAML config specifying your model, dataset, beta, and learning rate. A typical command is accelerate launch with scripts/run_simpo.py pointing at your training config.

What is the difference between SimPO and DPO?▼

SimPO is reference-free, optimizing directly on the policy model's average log probabilities with a target margin, while DPO requires a separate reference model to normalize log probability ratios. SimPO is simpler, uses less memory, and reports better performance than DPO.

What learning rate should I use for SimPO training?▼

For 7B-8B models, use 3e-7 to 5e-7; use 3e-7 for reasoning-intensive tasks like math and code. Larger models (70B+) need 5e-8 to 1e-7, while smaller models can tolerate up to 1e-6.

What GPU hardware does SimPO training require?▼

A 7B model trains on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model needs eight A100 80GB. BF16 mixed precision, gradient checkpointing, and Flash Attention 2 reduce memory usage.

Why does my SimPO training loss diverge?▼

Loss divergence usually means the learning rate or beta is too high. Reduce learning rate from 5e-7 to 3e-7 and beta from 2.0 to 1.0, then monitor the reward margin between chosen and rejected responses.

When should I use OpenRLHF or TRL instead of SimPO?▼

Use OpenRLHF when you need multi-node distributed training with PPO or GRPO, and TRL when you want multiple alignment methods in one framework. SimPO fits single-node preference training where simplicity and compute efficiency matter.

Related Skills

Explore More Education & Research Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
GitnappGitnapp

serving-llms-vllm

Deploy large language model inference services with vLLM and OpenAI-compatible endpoints.

Community
Advanced
SethyPagnaSethyPagna

fine-tuning-with-trl

Fine-tune language models with TRL for post-training alignment workflows.

Community
Advanced
Jensen-YaoJensen-Yao
2

openrlhf-training

Trains large language models with distributed PPO, GRPO, RLOO, and DPO using Ray and vLLM.

Community
Advanced