skills.rest
zerwiz
zerwizCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Optimize large language models from preference pairs without a reference model.

1|Updated Apr 10, 2026
Search Tags:#fine-tuning#hyperparameters#preference optimization#simpo#llm alignment#loss functions#training datasets
One-click install
npx skills add https://github.com/zerwiz/wayofpi --skill simpo-training-zerwiz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/zerwiz/wayofpi/tree/main/.hermes/hermes-agent/optional-skills/mlops/simpo
Command: npx skills add https://github.com/zerwiz/wayofpi --skill simpo-training-zerwiz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you align large language models with preference data without needing a reference model, making training simpler, faster, and more memory efficient than traditional DPO-style workflows.

Core Features & Use Cases

  • Preference optimization for base, instruct, and reasoning-focused models using chosen and rejected response pairs.
  • Practical guidance for selecting beta, gamma, learning rate, and SFT regularization settings across model sizes and task types.
  • Reference materials for dataset preparation, loss functions, hyperparameter tuning, and debugging training instability.
  • Use Case: You can fine-tune a chat model on preference pairs to improve helpfulness while preserving its existing instruction-following behavior.

Quick Start

Use this skill to configure a SimPO training run for your model, dataset, and optimization settings.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fine-tune an LLM with preference pairs without a reference model?▼

SimPO optimizes large language model behavior from preference pairs without needing a reference model, making training simpler, faster, and more memory efficient than traditional DPO-style workflows. It applies to base, instruct, and reasoning-focused models.

What hyperparameters do I need to configure for SimPO training?▼

SimPO training requires configurable beta and gamma margin settings, along with a learning rate. You can also apply optional SFT regularization to preserve existing instruction-following behavior while tuning for preference alignment.

How does SimPO preference optimization compare to DPO for chat alignment?▼

SimPO differs from DPO by eliminating the reference model requirement, which simplifies the training workflow and reduces memory overhead. It uses chosen and rejected response pairs with a length-normalized reward formulation and gamma margin settings.

Can I use SimPO for reasoning-heavy tasks like code generation and math tuning?▼

Yes, SimPO applies to reasoning-heavy tasks such as code generation, math tuning, and chat alignment. It optimizes model behavior for complex tasks using preference pairs and optional SFT regularization to maintain base capabilities.

What format does training data need to be in for SimPO fine-tuning?▼

SimPO requires stable training data formatting with chosen and rejected response pairs. The dataset must contain paired responses for each prompt to compute the preference optimization loss function across single-node or multi-GPU runs.

Related Skills

Explore More Software Engineering Skills→
thinking-machines-labthinking-machines-lab
4.0k

dpo

Configure and run Direct Preference Optimization training with a model and preference dataset.

Official
Intermediate
hochoa13hochoa13

simpo-training

Implements reference-free preference optimization for LLM alignment.

Community
Advanced
KappTech88KappTech88

simpo-training

Optimize LLM alignment using preference data without a reference model.

Community
Advanced
ihatesea69ihatesea69
3

simpo-training

Train LLMs with reference-free SimPO preference optimization.

Community
Advanced