simpo-training

Reference-free preference optimization aligns LLMs without a reference model by learning preferences directly. SimPO achieves this by using a specific configuration with beta, gamma_beta_ratio, sft_weight, and loss_type to optimize models from 7B to 70B.

SimPO is a reference-free preference optimization method that aligns LLMs without a ground-truth or reference model. It enables more scalable and efficient alignment by learning preferences directly from a base model and a preference dataset.

Yes, SimPO supports training models from 7B to 70B across single-node or distributed configurations. It is applicable to instruction-following, reasoning, and chat tasks using standard datasets.

To start SimPO training, install Python and PyTorch dependencies, prepare a base model and a preference dataset, configure the SimPO parameters, and run the recommended training script to begin reference-free optimization.

SFT regularization is optional in SimPO. You can configure it alongside the core SimPO parameters including beta, gamma_beta_ratio, and loss_type to balance preference learning with supervised fine-tuning.

Reference-free preference optimization eliminates the need for a reference model during LLM alignment, enabling more scalable and efficient training. This approach simplifies the pipeline while maintaining alignment for instruction-following and reasoning tasks.