skills.rest
ruiyangruiyi
ruiyangruiyiCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train large language models with reference-free SimPO preference alignment.

Updated May 26, 2026
Search Tags:#instruction tuning#preference optimization#simpo#accelerate#llm alignment#dpo alternative#preference datasets
One-click install
npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill simpo-training-ruiyangruiyi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/ruiyangruiyi/hermes-agent/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill simpo-training-ruiyangruiyi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve?

This Skill solves the problem of improving an LLM to follow preferences using chosen/rejected pairs, without requiring a reference model and with simpler, more efficient training than DPO-style methods.

Core Features & Use Cases

  • Reference-free preference optimization: Trains with SimPO directly from preference pairs by optimizing log-probability differences with a tunable target margin.
  • Config-driven training recipes: Supports practical workflows for base models and instruct models, including reasoning-focused settings and common hyperparameter adjustments.
  • Preference-alignment monitoring and troubleshooting: Covers typical issues like loss divergence, capability forgetting, poor preference separation, and OOM mitigation, with actionable parameter changes.

Quick Start

Train SimPO on the Mistral 7B base model using the provided SimPO training script and a matching configuration file for preference datasets.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with preference pairs without using a reference model?▼

SimPO trains preference-tuned LLMs directly from chosen/rejected pairs without a reference model, optimizing log-probability differences with a tunable target margin for efficient alignment.

What is the difference between SimPO and DPO for LLM alignment?▼

SimPO is a DPO alternative that eliminates the reference model requirement, offering simpler and more efficient training by optimizing log-probability differences with a tunable target margin.

How do I start SimPO training on a base model like Mistral 7B?▼

Launch SimPO training on a base model using the provided training script and matching configuration file with Accelerate, utilizing torch, transformers, datasets, and TRL dependencies.

Can I use SimPO for reasoning-focused instruction tuning?▼

Yes, SimPO supports reasoning-focused preference datasets and provides configurable training recipes for both base and instruct models, including general instruction tuning scenarios.

Why does my SimPO training loss diverge or cause capability forgetting?▼

SimPO training issues like loss divergence or capability forgetting can be mitigated by adjusting hyperparameters such as beta, gamma_beta_ratio, sft_weight, and loss_type.

How do I resolve out-of-memory errors during SimPO alignment training?▼

OOM mitigation during SimPO alignment involves actionable parameter changes and configuration adjustments within the Accelerate-based training environment to fit available memory.

Related Skills

Explore More Education & Research Skills→
JamesFincherJamesFincher

simpo-training

Train LLMs with reference-free SimPO preference optimization from chosen and rejected pairs.

Community
Advanced
Rawgrowth-ConsultingRawgrowth-Consulting

simpo-training

Automate reference-free preference optimization for LLM alignment with configurable loss functions.

Official
Advanced
zhouboyu-xrealzhouboyu-xreal

simpo-training

Optimizes LLM preferences with SimPO using torch, transformers, datasets, trl, and accelerate.

Community
Advanced
zhuangbiaoweizhuangbiaowei
2

simpo-training

Optimize LLM alignment with reference-free SimPO preference optimization.

Community
Advanced