simpo-training

Reference-free LLM alignment optimizes user-preference signals directly from chosen and rejected pairs. SimPO applies this technique to reduce training complexity by eliminating the fixed baseline requirement while maintaining efficient preference optimization.

SimPO is a reference-free alternative to DPO that aligns language models using win/lose preference pairs. Unlike DPO, it removes the reference model dependency, cutting training complexity while supporting configurable hyperparameters like beta and gamma.

Preference optimization training uses curated datasets of chosen and rejected responses to align LLMs. The process applies a configurable SimPO loss function with adjustable learning rates and optional SFT regularization to preserve model capabilities.

SimPO works with PyTorch, Transformers, Datasets, TRL, and Accelerate to optimize LLM preferences. It supports configurable training across various model sizes, allowing efficient alignment workflows without a reference model.

SFT regularization in reference-free preference optimization preserves the base model's existing capabilities during alignment. It prevents capability degradation when applying SimPO loss to optimize chosen and rejected pairs without a baseline.

SimPO requires curated chosen and rejected preference pairs for training. While it eliminates the reference model, hyperparameters like learning rate, beta, and gamma must be carefully configured to avoid degrading model capabilities across different model sizes.