simpo-training

Reference-free preference optimization trains language models directly using chosen and rejected preference pairs, removing the need for a separate reference model. It applies sigmoid or hinge loss with configurable beta and gamma margins to align base or instruct models efficiently.

SimPO fine-tuning simplifies LLM alignment by eliminating the reference model required in DPO. You select a preference dataset, configure hyperparameters like beta and target margin, and launch a training run with optional SFT regularization for stable results.

No, you do not need a reference model for this preference optimization. The SimPO approach trains language models directly on chosen and rejected preference pairs, reducing complexity and computational overhead for chat quality and reasoning alignment.

You configure SimPO hyperparameters by setting learning rate, beta, target margin, and gamma values. Adjusting these alongside loss-function selection between sigmoid and hinge helps ensure stable training and evaluation across different safety-alignment and reasoning scenarios.

Yes, you can apply optional SFT regularization during reference-free preference optimization. This helps preserve base model capabilities while aligning with chosen and rejected preference pairs for chat, reasoning, and safety tasks without relying on a reference model.

Preference optimization without a reference model requires chosen and rejected preference pairs. Proper dataset selection and preprocessing ensure the preference-pair formatting matches your alignment goals for chat quality, reasoning tasks, or safety fine-tuning.