Train language models with reference-free SimPO preference optimization.
npx skills add https://github.com/AVOI-CEO/avoi-agent --skill simpo-training-avoi-ceo
Please help me install this Agent Skill. Skill: simpo-training Source: https://github.com/AVOI-CEO/avoi-agent/tree/main/optional-skills/mlops/simpo Command: npx skills add https://github.com/AVOI-CEO/avoi-agent --skill simpo-training-avoi-ceo
SimPO provides a reference-free approach to aligning language models by optimizing preferences directly without requiring a reference model.
Install the required packages, prepare a preference dataset, and run the provided training script with a model configuration.
High-intent search queries and answers about installing and using this skill.
Reference-free LLM preference optimization trains models directly on preference data without a reference policy. SimPO eliminates the reference model requirement, reducing memory overhead while aligning instruction-following and chat models.
SimPO is a reference-free preference optimization approach for LLM alignment. It optimizes language models directly on preference datasets without requiring a reference policy, configurable via beta, gamma_beta_ratio, and loss_type parameters.
SimPO supports training language models from 7B to 70B parameters for instruction-following and chat alignment. It scales across model sizes using PyTorch, transformers, and accelerate, with optional gradient_checkpointing to manage memory.
SimPO training is configurable through parameters like learning_rate, beta, gamma_beta_ratio, loss_type, and label_smoothing. You can also enable SFT regularization via sft_weight to preserve base model capabilities during preference optimization.
SimPO uses a dataset_mixer configuration to combine multiple preference datasets for training. This allows you to blend various instruction-following and chat preference data sources within the trl and datasets framework.
Reference-free preference optimization avoids the computational cost of loading a reference model required by standard RLHF. SimPO achieves alignment directly, lowering memory requirements while maintaining configurable optimization through beta and loss_type settings.
MesferAliImplement SimPO training for LLM alignment using PyTorch and Transformers.
niznik-devOrchestrate SLURM fine-tuning jobs with torchtune and evaluate with inspect-ai.
Clay-HHKOptimize LLMs with reference-free preference alignment using sigmoid and hinge losses.
icyzhRun reference-free preference optimization on large language models.
