skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using HuggingFace alignment-handbook.

Updated Jun 7, 2026
Search Tags:#fine-tuning#HuggingFace#preference optimization#DPO alternative#simpo#llm alignment#reference-free
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill simpo-training-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill simpo-training-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a separate reference model, extra memory, and complex configuration. SimPO removes the reference model entirely while outperforming DPO, making preference alignment simpler and more compute-efficient. ## Core Features & Use Cases - Reference-Free Preference Training: Run SimPO training via HuggingFace alignment-handbook with DeepSpeed ZeRO-3 configs for models like Mistral 7B and Llama 3 8B. - Hyperparameter Guidance: Detailed references for beta, gamma-beta ratio, learning rate, and SFT weight selection by model size and task type. - Dataset Preparation: Guidance on preference datasets (UltraFeedback, HH-RLHF, math preference sets), quality filtering, mixing, and synthetic data generation. - Use Case: Fine-tune Llama 3 8B Instruct on cleaned UltraFeedback preferences with a single GPU node, adding light SFT regularization to preserve instruction-following ability. ## Quick Start Train Mistral 7B on the UltraFeedback preference dataset using SimPO with the default training configuration.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?▼

Install the HuggingFace alignment-handbook, then launch scripts/run_simpo.py with accelerate using a DeepSpeed ZeRO-3 config and a SimPO training YAML. The config specifies the model, preference dataset, beta, gamma-beta ratio, and learning rate.

SimPO vs DPO: which preference optimization method should I use?▼

SimPO is reference-free, requiring no reference model, and outperforms DPO by +6.4 points on AlpacaEval 2.0 while being more efficient. Use DPO when you need an established reference-model baseline or more conservative training.

What learning rate should I use for SimPO training on 7B models?▼

Use 3e-7 to 5e-7 for 7B-8B models, with 5e-7 as the standard default. For reasoning-intensive tasks like math or code, reduce to 3e-7 to avoid loss divergence and capability forgetting.

What datasets work with SimPO preference training?▼

Datasets need prompt, chosen, and rejected fields. Recommended options include HuggingFaceH4/ultrafeedback_binarized, argilla/ultrafeedback-binarized-preferences-cleaned, Anthropic/hh-rlhf, and argilla/distilabel-math-preference-dpo for math reasoning.

Why does SimPO training loss diverge and how do I fix it?▼

Loss divergence usually comes from a learning rate that is too high or excessive reward scaling. Reduce learning rate from 5e-7 to 3e-7 and lower beta from 2.0 to 1.0 to stabilize training.

What GPU hardware is required for SimPO training?▼

A 7B model needs one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model needs eight A100 80GB. BF16 mixed precision, gradient checkpointing, and Flash Attention 2 reduce memory usage.

Related Skills

Explore More Software Engineering Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
SethyPagnaSethyPagna

fine-tuning-with-trl

Fine-tune language models with TRL for post-training alignment workflows.

Community
Advanced
OpenLAIROpenLAIR
1.0k

openrlhf-training

Run distributed RLHF training for large language models on Ray clusters.

Official
Advanced