simpo-training

SimPO aligns LLMs without a reference model by using length-normalized reward targets to optimize preferences directly. You specify model_name_or_path, dataset_mixer, beta, and gamma_beta_ratio to launch a reference-free post-training run.

SimPO is a preference optimization method that eliminates the reference model required by DPO. This reference-free approach reduces memory overhead and simplifies deployment while achieving better alignment performance than DPO.

SimPO post-training requires setting core hyperparameters like beta, gamma_beta_ratio, learning_rate, sft_weight, and loss_type. You also specify optional settings such as num_train_epochs and per_device_train_batch_size to control the run.

SimPO supports preference-based fine-tuning across various model sizes with limited compute resources. By removing the reference model, it lowers deployment barriers and improves instruction-following efficiently.

SimPO does not require a reference model because it uses a reference-free optimization approach with length-normalized rewards. This mechanism directly aligns LLM outputs to preferences, bypassing the computational cost of loading a secondary model.

SimPO preference alignment requires specifying a dataset_mixer configuration to blend public datasets or custom preferences. This input feeds the reference-free optimization pipeline to fine-tune the target LLM.