skills.rest
JamesFincher
JamesFincherCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization from chosen and rejected pairs.

Updated May 4, 2026
Search Tags:#distributed training#preference optimization#simpo#llm alignment#dpo alternative#reference-free#chosen rejected training
One-click install
npx skills add https://github.com/JamesFincher/gengar --skill simpo-training-jamesfincher
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/JamesFincher/gengar/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/JamesFincher/gengar --skill simpo-training-jamesfincher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

SimPO training addresses the challenge of aligning an LLM to human preferences efficiently without needing a reference model or complex RL infrastructure.

Core Features & Use Cases

  • Reference-free preference optimization: Trains on chosen/rejected pairs using SimPO, avoiding the extra forward-pass and maintenance burden of a separate reference model.
  • Preference-alignment with tunable objectives: Supports adjustable reward scaling (beta), target margin (gamma_beta_ratio), optional SFT regularization (sft_weight), and loss types (sigmoid/hinge).
  • Practical training workflows: Enables end-to-end preference fine-tuning for base models and instruct/chat models, including dataset mixing for domains like general help, math reasoning, and safety/harmlessness.

Use Case Example: You have an instruction-tuning model and a dataset of (prompt, chosen response, rejected response) pairs; use SimPO to improve helpfulness and adherence to preferences while controlling capability drift via SFT regularization when needed.

Quick Start

Ask the agent to train a Mistral 7B SimPO model from the base checkpoint using HuggingFaceH4/ultrafeedback_binarized with learning_rate=5e-7, beta=2.0, gamma_beta_ratio=0.5, loss_type=sigmoid, sft_weight=0.0, and outputs saved to ./outputs/mistral-7b-simpo.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I align an LLM with preference data without a reference model?▼

SimPO training aligns an LLM with preference data without a reference model by processing chosen and rejected pairs directly. It avoids the extra forward-pass and maintenance burden of a separate reference model.

What is reference-free preference optimization for large language models?▼

Reference-free preference optimization is an alignment technique that trains large language models on chosen and rejected pairs without comparing against a baseline model. SimPO applies this to improve instruction following and domain-specific reasoning.

What hyperparameters do I need to configure for SimPO training?▼

SimPO training requires configuring beta for reward scaling, gamma_beta_ratio for target margin, loss_type for sigmoid or hinge calculation, and an optional sft_weight to control capability drift through SFT regularization.

Do I need trl and accelerate to run SimPO training workflows?▼

Yes, you need trl and accelerate alongside torch, transformers, and datasets to establish a compatible training environment for executing SimPO preference optimization workflows.

How does SimPO compare to DPO for instruction tuning?▼

SimPO improves upon DPO for instruction tuning by operating reference-free, removing the need for an additional reference model forward-pass during training while still optimizing helpfulness and adherence to human preferences.

Can I use SFT regularization to prevent capability drift during preference optimization?▼

Yes, you can use SFT regularization to prevent capability drift during preference optimization by adjusting the sft_weight parameter, which anchors the model to its original instruction-tuning capabilities while learning new preferences.

Related Skills

Explore More Education & Research Skills→
ruiyangruiyiruiyangruiyi

simpo-training

Train large language models with reference-free SimPO preference alignment.

Community
Intermediate
ScientiaCapitalScientiaCapital
1

unsloth-finetuning

Fine-tune Llama, Mistral, Phi, or Gemma models with LoRA and 4-bit quantization.

Community
Intermediate
MesferAliMesferAli

simpo-training

Implement SimPO training for LLM alignment using PyTorch and Transformers.

Community
Advanced
Chris-Chai-MinjaeChris-Chai-Minjae

fine-tuning-with-trl

Fine-tune LLMs with TRL using SFT, DPO, PPO, and GRPO.

Community
Advanced