What problem does it solve?
This Skill solves the problem of aligning an LLM to human preferences by training on chosen-versus-rejected responses without requiring a separate reference model, reducing setup cost while improving preference performance.
Core Features & Use Cases
- Reference-free preference optimization: Implements SimPO as a DPO alternative that does not need π_ref, making training simpler and more efficient.
- Drop-in training workflows: Provides practical end-to-end guidance for configuring and launching SimPO runs on base models and instruct models using Accelerate + DeepSpeed ZeRO-3.
- Preference-alignment controls: Covers key hyperparameters such as beta, gamma_beta_ratio, loss_type (sigmoid/hinge), and sft_weight to balance alignment strength vs capability retention.
- Debugging and tuning guidance: Diagnoses common issues like loss divergence, forgetting, poor preference separation, and OOM by adjusting learning rate, beta, margins, and SFT regularization.
Use cases:
- Align a base LLM to follow instructions better using general preference datasets.
- Fine-tune an instruct/chat model for preference alignment while minimizing capability regression by adding SFT regularization.
- Optimize math/code reasoning alignment by using lower learning rates and stronger margin settings.
Quick Start
Use the simpo-training skill to train SimPO on Mistral 7B by running accelerate launch with the deepspeed_zero3 config and a SimPO training config YAML.
