simpo-training

SimPO trains an LLM from chosen-versus-rejected preference pairs without using a reference model. It applies to instruction alignment and preference optimization across base and instruct/chat models, including general and math/reasoning datasets. It satisfies reference-free SimPO objectives by optimizing beta-scaled log-probability ratios with an optional SFT regularization term using a configurable loss type and margins.

SimPO is a reference-free alternative to DPO, meaning it does not require a separate reference model during training. This reduces extra forward-pass complexity while still pushing the model toward chosen answers over rejected ones.

Yes, SimPO applies to instruction alignment and preference optimization across both base and instruct/chat models. It supports general datasets as well as math and reasoning datasets using chosen and rejected data pairs.

You can tune training stability by configuring beta, gamma/beta margin, loss type (sigmoid or hinge), and an optional SFT regularization term. These controls help reduce divergence and capability loss during preference optimization.

Generate a SimPO training configuration for your base model and preference dataset, then launch training using the provided accelerate Deepspeed ZeRO-3 setup to fine-tune instruct models efficiently on preference pairs.

Add optional SFT regularization during SimPO training when you want to reduce divergence and prevent capability loss. It acts alongside the beta-scaled log-probability ratios to maintain stable chosen-versus-rejected behavior alignment.