simpo-training

You can align LLM preferences without a reference model by using SimPO, which trains directly from chosen versus rejected response pairs to improve preference separation and avoid the added complexity of reference-based methods.

Reference-free preference optimization is a technique that aligns LLMs using chosen versus rejected response pairs directly, eliminating the need to train or load a separate reference model during the alignment process.

Run preference alignment training by executing accelerate launch with scripts/run_simpo.py and a YAML config file, applying SimPO hyperparameters like beta, gamma_beta_ratio, loss_type, and optional sft_weight.

Yes, SimPO supports single-node distributed setups configured via accelerate, allowing you to integrate DeepSpeed ZeRO-3 to optimize memory and compute when running preference optimization on large language models.

SimPO offers faster iteration compared with DPO setups when compute is limited, because it skips training a separate reference model while still using chosen versus rejected pairs for preference alignment.

You use the optional sft_weight parameter to add SFT regularization, which reduces forgetting and helps preserve capabilities in instruct or chat models during the reference-free preference alignment process.