simpo-training

Reference-free preference optimization aligns LLMs by learning preferences directly from on-policy signals without loading a separate reference policy, reducing memory requirements. It applies deterministic training workflows with configurable parameters like beta and loss_type.

To fine-tune instruction-following models like Mistral-7B or Llama-3-8B, prepare your preference dataset and run the training script. The workflow supports YAML-configurable settings with model-size-specific defaults to streamline the alignment process.

No, you do not need a reference model for this DPO alternative. It optimizes preferences directly from on-policy signals, which makes fine-tuning more efficient than standard DPO or PPO workflows that require loading a separate reference policy.

You can configure beta, gamma_beta_ratio, loss_type, and sft_weight for your training runs. These settings are managed through YAML-configurable files and include model-size-specific defaults to ensure optimal preference alignment.

Yes, reference-free alignment works directly with transformers and trl. The training workflow explicitly integrates these dependencies alongside torch, datasets, and accelerate to execute the fine-tuning process for instruction-following models.

SimPO provides higher efficiency than PPO by eliminating the need for a separate reference model during preference optimization. It learns directly from on-policy signals, reducing computational overhead while maintaining deterministic, configurable training.