simpo-training

Reference-free preference optimization trains an LLM directly using chosen versus rejected response pairs to learn preference-aligned behavior, avoiding the extra reference-model setup required by DPO. It uses policy log-probabilities with margin control parameters like beta and gamma_beta_ratio.

SimPO is a simpler alternative to DPO that aligns chat models efficiently using chosen and rejected preference pairs. It trains directly from policy log-probabilities, eliminating the need to maintain a reference model while still improving evaluation performance.

To configure SimPO training, you use beta for reward scaling and gamma_beta_ratio to set a target margin that controls how strongly preferences are enforced. You also specify the loss_type and run the training entrypoint with a compatible dataset_mixer and dataset splits.

Yes, SimPO supports common preference datasets and typical distributed training setups, including DeepSpeed ZeRO-3 via Accelerate. This allows efficient reference-free preference optimization across distributed environments for instruction tuning and chat alignment.

You can prevent capability loss during preference optimization by applying optional SFT regularization. This works alongside reference-free SimPO training and margin control parameters like beta and gamma_beta_ratio to maintain the model's original abilities while enforcing preferences.