Optimize LLM preference alignment without a reference model using comparison data.
npx skills add https://github.com/ovachiever/droid-tings --skill simpo-training
Please help me install this Agent Skill. Skill: simpo-training Source: https://github.com/ovachiever/droid-tings/tree/main/skills/simpo Command: npx skills add https://github.com/ovachiever/droid-tings --skill simpo-training
This Skill describes SimPO, a reference-free preference optimization method to align LLMs, offering stronger performance than DPO while eliminating the need for a reference model.
Launch a SimPO training job for a base model (e.g., Mistral-7B) with a small preference dataset and monitor reward margins.
High-intent search queries and answers about installing and using this skill.
SimPO is a reference-free preference optimization method that delivers stronger alignment performance than DPO while eliminating the need for a reference model, reducing computational overhead compared to PPO-based approaches.
Yes. SimPO enables reference-free preference optimization by computing loss directly from user-provided comparison data without requiring a separate reference policy, making alignment training more resource-efficient.
SimPO supports UltraFeedback, Argilla UltraFeedback, Distilabel Math, and HelpSteer datasets for training base models, fine-tuning instruct models, and handling reasoning-intensive tasks.
SimPO provides configurable sigmoid or hinge loss options with tunable beta and gamma_beta_ratio parameters, plus SFT regularization to control preference alignment strength and margin behavior.
Yes. SimPO is compatible with single-node accelerated training configurations and supports learning-rate, batch-size, and gradient-accumulation tuning for efficient resource utilization.
SimPO aligns base LLMs, instruct models, and chat models using preference comparison data, with versatile workflows supporting different model types and training objectives.
thinking-machines-labConfigure and run Direct Preference Optimization training with a model and preference dataset.
rocklambrosAudit RLHF, DPO, and RLAIF checkpoints for reward hacking by comparing reward win-rates with preference win-rates.
frankxaiFine-tune large language models with HuggingFace TRL, Transformers, and PEFT.
GarrettRoiFine-tune LLMs with TRL using SFT, DPO, PPO, and GRPO.
