simpo-training

Reference-free preference optimization aligns large language models directly from chosen/rejected preference data without needing a separate reference policy. SimPO enables this by applying direct policy improvements, bypassing the overhead of loading and querying a reference model during training.

You train a Mistral-7B model with UltraFeedback by configuring SimPO's training dynamics on the preference dataset. You adjust parameters like beta, gamma_beta_ratio, and loss_type, optionally adding SFT regularization to preserve capabilities while learning from chosen and rejected pairs.

Yes, you can optimize 70B scale models without a reference model. SimPO supports flexible model scaling from 7B to 70B, applying direct policy improvements from preference data while removing the memory and compute bottleneck of running a secondary reference policy.

The best way to preserve capabilities during preference optimization is enabling SFT regularization. SimPO allows you to configure an sft_weight parameter alongside beta and gamma_beta_ratio, balancing preference learning with supervised fine-tuning to maintain general instruction following and reasoning.

Preference optimization works for coding and reasoning tasks alongside general instruction following. SimPO applies direct policy improvements from preference data across these domains, allowing you to align base models or instruct variants for specific task performance without a reference model.

SimPO does not require a reference model because it computes direct policy improvements from the chosen and rejected preference pairs alone. This reference-free approach eliminates the need to load a secondary model, reducing memory requirements while still enabling effective alignment from datasets like UltraFeedback.