Optimize language model preference alignment with SimPO training.
npx skills add https://github.com/JKhyro/HERMES-AGENT --skill simpo-training-jkhyro
Please help me install this Agent Skill. Skill: simpo-training Source: https://github.com/JKhyro/HERMES-AGENT/tree/main/optional-skills/mlops/simpo Command: npx skills add https://github.com/JKhyro/HERMES-AGENT --skill simpo-training-jkhyro
This Skill helps you align language models with preference data using SimPO, removing the need for a reference model while keeping training simpler and more efficient than DPO or PPO.
Use the simpo-training skill to produce a SimPO training plan or configuration for your target model, dataset, and alignment goal.
High-intent search queries and answers about installing and using this skill.
SimPO aligns language models using chosen and rejected preference pairs without maintaining a separate baseline model. It simplifies preference optimization by removing the reference model requirement while keeping training stable through beta, gamma-beta ratio, and SFT weight tuning.
SimPO is a reference-free alternative to DPO that eliminates the need for a separate reference model during preference optimization. It simplifies the training workflow and reduces memory overhead while achieving comparable alignment results using sigmoid or hinge loss functions.
Preference datasets for SimPO training must be formatted as chosen-rejected pairs. The Skill guides proper dataset formatting for general chat, reasoning, and math preference datasets, ensuring the reference-free loss function can correctly compare and optimize model outputs.
Yes, SimPO supports compact single-node workflows for fine-tuning 7B or 8B chat models. It applies reference-free loss selection and SFT regularization to improve instruction following, reasoning, or math performance within a single-machine training setup.
Stable SimPO alignment requires tuning learning rates, beta, the gamma-beta ratio, and the SFT regularization weight. Proper configuration of these hyperparameters ensures the reference-free preference optimization process remains stable and effective.
SimPO supports both sigmoid and hinge loss for reference-free preference optimization. The choice depends on your specific alignment goals and dataset characteristics, with the Skill providing practical guidance on selecting the appropriate loss function for your training configuration.
SethyPagnaTrain language models with SimPO preference alignment without a reference model.
ScientiaCapitalExport fine-tuned models to GGUF and deploy with vLLM or Ollama.
zerwizOptimize large language models from preference pairs without a reference model.
richardnguyen0715Optimize language model alignment without a reference model using SimPO.
