simpo-training

Reference-free preference optimization aligns LLMs by optimizing preferred responses without a reference model. SimPO implements this using a length-normalized reward and a gamma-beta ratio, enabling efficient post-training alignment.

You can align an LLM without a reference model by applying SimPO. It uses a reference-free loss function with configurable beta and gamma-beta ratios to optimize preferred choices directly from your dataset.

Yes, SimPO works with PyTorch and Hugging Face transformers. It relies on standard dependencies including torch, transformers, datasets, trl, and accelerate to run configurable training across common model sizes.

SimPO removes the need for a separate reference model required in standard RLHF. It implements a reference-free loss with optional SFT regularization, reducing memory and computation overhead during LLM alignment.

Yes, you can add optional SFT regularization during SimPO training. This helps preserve the model's original capabilities and prevents capability degradation while applying reference-free preference optimization.

Dataset preparation for SimPO involves formatting preference data with chosen and rejected responses. The skill supports flexible workflow configurations to prepare datasets and run training across common LLM sizes.