simpo-training

Reference-free LLM preference optimization aligns models by directly rewarding chosen outputs over rejected ones. SimPO implements this by eliminating the fixed reference model, using configurable parameters like beta and loss_type to optimize instruction-following and reasoning tasks directly from preference datasets.

To train an LLM with a preference dataset without a reference model, provide a base model, labeled chosen/rejected outputs, and training configuration parameters like learning_rate, beta, gamma_beta_ratio, and loss_type. SimPO applies optional SFT regularization to preserve base capabilities during alignment.

SimPO works for code generation, reasoning, and general instruction-following across various model sizes. Its flexible workflow supports hyperparameter tuning with configurable beta, gamma_beta_ratio, and loss_type to adapt preference optimization to different task requirements.

The best way to align LLMs without a separate reference model is using SimPO, which directly optimizes preferences from chosen vs rejected outputs. This approach reduces deployment overhead by removing the reference model while applying configurable loss_type and SFT regularization for stable training.

SFT regularization is needed during preference optimization to preserve the base model's existing capabilities. SimPO applies optional SFT regularization alongside preference learning to prevent capability degradation when directly rewarding chosen outputs over rejected ones during alignment training.

SimPO training requires configuring learning_rate, beta, gamma_beta_ratio, and loss_type parameters. You also need a base model and labeled preference dataset with chosen and rejected outputs, with optional SFT regularization to control the alignment process across instruction-following and reasoning tasks.