simpo-training

SimPO is a reference-free preference optimization method that aligns LLMs using chosen and rejected pairs, eliminating the need for a separate reference policy required by DPO. It directly optimizes models using sigmoid or hinge loss.

Fine-tune a 7B model on UltraFeedback pairs by applying SimPO with configurable beta, gamma_beta_ratio, and SFT regularization. Use Hugging Face datasets and accelerate configurations for efficient training on single-node or small multi-GPU setups.

Yes, SimPO integrates natively with PyTorch, Hugging Face datasets, trl, accelerate, and DeepSpeed. This integration supports efficient preference optimization training across single-node and small multi-GPU environments.

Yes, SimPO applies to fine-tuning base or instruct models on paired preference datasets for chat, code, and math reasoning tasks. It serves as a DPO alternative by removing the reference model to reduce compute and complexity.

SimPO exposes configurable training controls including beta, gamma_beta_ratio, loss_type (sigmoid or hinge), sft_weight, learning rate, and batch accumulation settings. These parameters allow model-size-specific tuning for preference optimization.

DPO requires loading and querying a separate reference model alongside the target model during training. SimPO removes this reference policy overhead, directly reducing memory usage and compute complexity for preference alignment.