simpo-training

Name: simpo-training
Availability: InStock
Author: eddielueng

Train LLMs with reference-free SimPO using chosen versus rejected preference pairs.

1|Updated Apr 14, 2026

Search Tags:#hyperparameter tuning #distributed training #preference optimization #simpo #llm alignment #dpo alternative #chosen rejected

Ready to run

npx skills add https://github.com/eddielueng/hermes-agent-zh --skill simpo-training-eddielueng

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/eddielueng/hermes-agent-zh/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/eddielueng/hermes-agent-zh --skill simpo-training-eddielueng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

SimPO solves the problem of efficiently aligning LLMs to human or evaluator preferences using chosen/rejected response pairs without requiring a reference model.

Core Features & Use Cases

Reference-free Preference Optimization: Trains directly from policy log-probabilities for chosen vs rejected outputs, avoiding the extra reference-model setup used by DPO.
Efficient Alignment with Margin Control: Uses beta for reward scaling and gamma_beta_ratio for a target margin to control how strongly preferences are enforced.
Practical Training Workflows: Supports common preference datasets and typical distributed training setups (e.g., DeepSpeed ZeRO-3 via Accelerate), plus optional SFT regularization to reduce capability loss.
Use Case: You have preference data (chosen/rejected pairs) and want a faster, simpler alternative to DPO that still improves evaluation performance for instruction/chat models.

Quick Start

Start SimPO training for a Mistral 7B preference model using the provided training script and a matching SimPO YAML config.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How does reference-free preference optimization work for LLM alignment?▼

Reference-free preference optimization trains an LLM directly using chosen versus rejected response pairs to learn preference-aligned behavior, avoiding the extra reference-model setup required by DPO. It uses policy log-probabilities with margin control parameters like beta and gamma_beta_ratio.

What is the best way to align a chat model without training a separate reference model?▼

SimPO is a simpler alternative to DPO that aligns chat models efficiently using chosen and rejected preference pairs. It trains directly from policy log-probabilities, eliminating the need to maintain a reference model while still improving evaluation performance.

How do I configure hyperparameters like beta and gamma_beta_ratio for SimPO training?▼

To configure SimPO training, you use beta for reward scaling and gamma_beta_ratio to set a target margin that controls how strongly preferences are enforced. You also specify the loss_type and run the training entrypoint with a compatible dataset_mixer and dataset splits.

Does SimPO work with distributed training setups like DeepSpeed ZeRO-3?▼

Yes, SimPO supports common preference datasets and typical distributed training setups, including DeepSpeed ZeRO-3 via Accelerate. This allows efficient reference-free preference optimization across distributed environments for instruction tuning and chat alignment.

How can I prevent capability loss during preference optimization?▼

You can prevent capability loss during preference optimization by applying optional SFT regularization. This works alongside reference-free SimPO training and margin control parameters like beta and gamma_beta_ratio to maintain the model's original abilities while enforcing preferences.