simpo-training

Reference-free preference optimization is an alignment method that trains large language models directly on user preference data without requiring a separate reference model. SimPO implements this technique to reduce compute overhead while improving instruction-following capabilities.

SimPO training is configured using parameters like learning_rate, beta, gamma_beta_ratio, sft_weight, loss_type, and dataset_mixer. These control training dynamics, evaluation, and regularization for aligning models of varying sizes on preference datasets.

Yes, SimPO works with public datasets such as Ultrafeedback and Distilabel-Math. It applies reference-free preference optimization to instruction-following and math reasoning tasks, supporting configurable model sizes and efficient compute budgets.

SimPO eliminates the need for a separate reference model during preference optimization, significantly reducing compute requirements. It outperforms baseline methods for instruction-following tasks while relying solely on preference data for alignment.

Yes, SimPO supports math reasoning and general alignment scenarios. It applies reference-free optimization to both instruction-following and reasoning tasks using configurable parameters like loss_type and sft_weight for regularization.