skills.rest
ZardLi1115
ZardLi1115COMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train LLMs from chosen/rejected preference pairs using SimPO without a reference model.

4|Updated May 18, 2026
Search Tags:#preference optimization#simpo#llm alignment#dpo alternative#chosen rejected#accelerate training#preference datasets
One-click install
npx skills add https://github.com/ZardLi1115/zedclaw --skill simpo-training-zardli1115
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/ZardLi1115/zedclaw/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/ZardLi1115/zedclaw --skill simpo-training-zardli1115

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve?

SimPO addresses the need to align LLM behavior using chosen/rejected preference pairs while avoiding the complexity and extra compute of reference-model-based methods like DPO.

Core Features & Use Cases

  • Reference-free preference optimization: Trains directly from (prompt, chosen, rejected) pairs without requiring a separate reference model.
  • Efficient DPO alternative: Uses SimPO’s optimized objective to achieve strong preference separation with fewer forward-pass requirements.
  • Control alignment strength: Tunes key hyperparameters such as beta and gamma/beta to adjust margin and alignment aggressiveness.

Use case example: You have preference data for instruction following (chosen/rejected responses) and want to fine-tune a 7B or 8B model for better preference alignment using a simpler, faster workflow than DPO.

Quick Start

Use the simpo-training skill to train your model on preference pairs by supplying your target model, dataset_mixer, and SimPO hyperparameters from the provided configuration examples.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with preference pairs without using a reference model?▼

SimPO trains LLMs directly from chosen and rejected preference pairs without a reference model, reducing compute complexity. It uses an optimized objective to achieve strong preference separation while avoiding the extra forward passes required by DPO.

What is the difference between SimPO and DPO for LLM alignment?▼

SimPO is a reference-free alternative to DPO that eliminates the need for a separate reference model during preference optimization. This reduces memory and compute requirements while still aligning model behavior using chosen and rejected response pairs.

Can I use accelerate and DeepSpeed for distributed preference optimization training?▼

Yes, SimPO supports distributed training settings via accelerate and DeepSpeed. You can configure single-node preference fine-tuning for 7B or 8B models by specifying appropriate distributed training settings alongside SimPO hyperparameters.

How do I adjust alignment strength during reference-free preference optimization?▼

You control alignment strength by tuning key SimPO hyperparameters such as beta, gamma_beta_ratio, learning_rate, loss_type, and sft_weight. These parameters adjust the margin and aggressiveness of the preference separation.

What format does preference data need to be in for SimPO training?▼

SimPO requires preference datasets structured as chosen and rejected pairs, configured via dataset_mixer and dataset_splits. This data is used to optimize the model for instruction following, chat, coding, or math reasoning tasks.

Do I need a separate SFT step before applying SimPO to my language model?▼

SimPO includes an sft_weight hyperparameter to balance preference optimization with supervised fine-tuning. You can apply it directly to your target model using transformers and torch to streamline the alignment workflow.

Related Skills

Explore More Education & Research Skills→
thinking-machines-labthinking-machines-lab
4.0k

dpo

Configure and run Direct Preference Optimization training with a model and preference dataset.

Official
Intermediate
handsomelong922handsomelong922

model-merging

Merge fine-tuned models into a single hybrid using YAML-configured methods.

Community
Advanced
huggingfacehuggingface
10.9k

hugging-face-model-trainer

Train and fine-tune language models on Hugging Face Jobs using TRL methods.

Official
Advanced
Orchestra-ResearchOrchestra-Research
11.5k

simpo-training

Train large language models with SimPO preference optimization without a reference model.

Official
Intermediate