simpo-training

Reference-free LLM alignment optimizes language models without a baseline model. SimPO works by directly maximizing the difference in log-probabilities between chosen and rejected responses from preference datasets to efficiently fine-tune model behavior.

You can fine-tune a language model by applying the SimPO loss formulation to your preference dataset. Configure hyperparameters like learning rate, beta, and gamma_beta_ratio, then launch a training run on your chosen and rejected response pairs.

Key hyperparameters for preference learning include learning rate, beta, gamma_beta_ratio, and loss type. You can also enable optional SFT regularization to preserve the base model's existing capabilities during the alignment process.

Yes, SimPO supports fine-tuning instruction-following and chat models on domain-specific data. You can utilize offline preference datasets like Ultrafeedback or Distilabel Math, and apply optional dataset mixing and data preprocessing steps.

Reference-free preference optimization mitigates capability loss by applying optional SFT regularization. This adds a supervised fine-tuning loss term alongside the chosen and rejected log-probability difference to preserve the model's original performance.