What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a reference model or reward model, adding memory cost and complexity. SimPO removes the reference model entirely while outperforming DPO, making preference alignment simpler and more compute-efficient. ## Core Features & Use Cases - Reference-Free Preference Training: Train models like Mistral 7B or Llama 3 8B on chosen/rejected preference pairs using the alignment-handbook with DeepSpeed ZeRO-3 and Flash Attention 2. - Hyperparameter Guidance: Detailed references for beta, gamma-beta ratio, learning rate, SFT weight, and loss type (sigmoid vs hinge) tuned by model size and task type. - Dataset Preparation: Guidance on preference data formats, quality filtering, deduplication, and mixing datasets like UltraFeedback, HH-RLHF, and math preference sets. - Use Case: Fine-tune Llama 3 8B Instruct on cleaned UltraFeedback preferences with beta 2.5 and a small SFT weight to improve AlpacaEval performance without a reference model. ## Quick Start Train Mistral 7B with SimPO on the UltraFeedback preference dataset using the provided training configuration and accelerate launch command.
