skills.rest
agtktID
agtktIDCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Trains language models with reference-free SimPO preference optimization using preference pair datasets.

1|Updated Aug 6, 2026
Search Tags:#fine-tuning#HuggingFace#preference optimization#DPO alternative#simpo#post-training#llm alignment
One-click install
npx skills add https://github.com/agtktID/indagis-agent --skill simpo-training-agtktid
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/agtktID/indagis-agent/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/agtktID/indagis-agent --skill simpo-training-agtktid

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning language models to human preferences typically requires DPO or PPO setups with reference models and reward models, adding memory cost and complexity. SimPO removes the reference model entirely while matching or exceeding DPO performance. ## Core Features & Use Cases - Reference-Free Preference Training: Run SimPO training on chosen/rejected preference pairs with beta, gamma-beta ratio, and optional SFT regularization via HuggingFace alignment-handbook configs. - Workflow Templates: Pre-built configurations for base models (Mistral 7B), instruct models (Llama 3 8B), and reasoning-intensive tasks like math and code. - Troubleshooting & Tuning Guides: Reference docs covering loss functions, hyperparameter selection by model size, and preference dataset preparation. - Use Case: Fine-tune Llama 3 8B Instruct on UltraFeedback preference data with a single accelerate launch command and DeepSpeed ZeRO-3, without loading a separate reference model. ## Quick Start Train Mistral 7B on the UltraFeedback preference dataset using SimPO with the provided training configuration.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?▼

Install the HuggingFace alignment-handbook, then launch training with accelerate using a SimPO YAML config specifying your model, preference dataset, beta, and gamma_beta_ratio. A typical command is accelerate launch with a DeepSpeed ZeRO-3 config and scripts/run_simpo.py.

SimPO vs DPO: which preference optimization method should I use?▼

SimPO is simpler and reference-free, eliminating the reference model required by DPO while achieving better performance on preference benchmarks. Use DPO when you need an established baseline comparison or more conservative training close to a reference policy.

What dataset format does SimPO training require?▼

SimPO requires preference pairs with prompt, chosen, and rejected fields, such as HuggingFaceH4/ultrafeedback_binarized or Anthropic/hh-rlhf. Alternative field names like question, winner, and loser are auto-detected by the training pipeline.

What learning rate should I use for SimPO on a 7B model?▼

Use 3e-7 to 5e-7 for 7B models, with 5e-7 as the standard default for general chat tasks. Lower it to 3e-7 for math or code reasoning tasks, and reduce further if the loss diverges during training.

Why does my SimPO model forget its original capabilities?▼

Catastrophic forgetting happens when pure preference optimization overwrites instruct or chat abilities. Add SFT regularization by setting sft_weight to 0.1-0.2, which mixes cross-entropy loss on chosen responses into the total loss.

What GPU hardware is needed for SimPO training?▼

A 7B model fits on one A100 40GB with DeepSpeed ZeRO-3, while 8B models need two A100 40GB and 70B models need eight A100 80GB. Enable BF16 mixed precision, gradient checkpointing, and Flash Attention 2 to reduce memory usage.

Related Skills

Explore More Software Engineering Skills→
atrawogatrawog

dpo

Optimize policy directly from chosen/rejected preference pairs using DPOTrainer and DPOConfig.

Community
Advanced
NVIDIANVIDIA
3.7k

evaluation

Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher.

Official
Advanced
DifanaDAPDifanaDAP

fine-tuning-with-trl

Train and align language models with human preferences using TRL techniques.

Community
Advanced
adm-humanerdadm-humanerd

fine-tuning-with-trl

Automate end-to-end RLHF fine-tuning of LLMs with TRL.

Community
Advanced