simpo-training

Reference-free preference optimization aligns LLMs by directly optimizing preferred responses, removing the need for a separate reference policy. SimPO uses configurable hyperparameters like beta and gamma to balance alignment and capability preservation.

Yes, the training pipeline uses transformers, datasets, torch, trl, and accelerate to run the preference optimization workflow. You provide a chosen model and preference dataset to start the SimPO training process.

SimPO is a reference-free approach, meaning it directly learns preferred responses without requiring a reference model during training. It also supports optional SFT regularization to balance capability preservation and alignment.

You can configure beta, gamma, and SFT weighting to balance performance and capability preservation for instruction following and reasoning tasks. These hyperparameters are modular and adjustable across various model sizes.

You should avoid reference-free preference optimization if your alignment task strictly requires a reference policy for probability calibration, or if you lack a properly structured preference dataset of chosen and rejected responses.