simpo-training

Reference-free preference optimization aligns language models using preference data without requiring a separate reward model. SimPO applies this technique by utilizing configurable hyperparameters like beta and gamma_beta_ratio to control reward strength and margins, streamlining the training process.

You can train a model with SimPO by installing dependencies like PyTorch, transformers, and accelerate, then running the provided training script with your chosen YAML configuration. The YAML files manage model selection and training parameters like learning rate and sft_weight.

Yes, SimPO supports reference-free optimization for aligning various base models ranging from 7B to 70B parameters. It works across single-node or multi-node setups using common datasets and YAML configurations for scalable instruction-following improvements.

SimPO training relies on hyperparameters such as beta for reward strength, gamma_beta_ratio for margin, and sft_weight for regularization. These configurable parameters help tailor alignment quality while preserving the base model's existing capabilities.

No, SimPO is a reference-free alignment method that does not require a separate reward model. It enables efficient, model-agnostic preference learning directly from preference data workflows including train/test splits and synthetic data generation.