simpo-training

Reference-free preference optimization is an LLM alignment method that trains models on human preferences without requiring a separate reference model. SimPO implements this approach to reduce compute costs compared to DPO or PPO while maintaining high alignment performance.

To train a Llama 3 model using SimPO, you run ready-to-use training scripts with a preference dataset using Python, PyTorch, Transformers, Datasets, TRL, and Accelerate. You can configure hyperparameters to execute the training process for 7B to 70B parameter models.

Yes, SimPO supports preference optimization for 7B to 70B parameter models across general chat, code, and reasoning tasks. You need a preference dataset and standard training frameworks like PyTorch and TRL to execute the training scripts.

SimPO offers a simpler, reference-free method for aligning large language models to human preferences, eliminating the need for a separate reference model. This approach reduces compute costs compared to DPO or PPO while delivering higher alignment performance.

No, you do not need a reference model to perform LLM alignment with SimPO. It is specifically designed as a reference-free preference optimization method, which eliminates the extra model requirement and reduces overall training compute costs.

Preference optimization training failures on datasets like UltraFeedback often relate to hyperparameters or dataset preparation. SimPO provides guidance on hyperparameter configuration, dataset preparation, and troubleshooting to resolve common training issues.