simpo-training

SimPO aligns LLMs without a reference model by optimizing preferences directly from chosen and rejected data. It reduces memory overhead by eliminating the reference model, enabling efficient preference optimization for diverse instruction-following tasks.

You can fine-tune models using Ultrafeedback-style data by configuring the dataset_mixer in your YAML file. Specify model_path, learning_rate, beta, and sft_weight to train preferences directly with torch and trl without a reference model.

Yes, reference-free preference optimization fully integrates with torch and transformers. It leverages the trl and accelerate libraries to manage training loops and distributed execution across model sizes and diverse preference datasets.

SimPO provides a reference-free alternative to DPO by eliminating the separate reference model during preference optimization. This reduces memory requirements while allowing configurable SFT regularization through sft_weight and gamma_beta_ratio parameters.

Yes, you can configure SFT regularization during preference training by adjusting the sft_weight parameter in your YAML configuration. This balances reference-free preference optimization with supervised fine-tuning objectives across your chosen datasets.