skills.rest
Z43L
Z43LCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Optimize LLM policies on chosen versus rejected pairs with SimPO.

Updated May 5, 2026
Search Tags:#distributed training#preference optimization#simpo#llm alignment#dpo alternative#chosen rejected#sft regularization
One-click install
npx skills add https://github.com/Z43L/zeus-agent --skill simpo-training-z43l
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/Z43L/zeus-agent/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/Z43L/zeus-agent --skill simpo-training-z43l

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve?

SimPO solves the problem of aligning large language models to preference data (chosen vs. rejected responses) without the extra complexity of training or maintaining a reference model.

Core Features & Use Cases

  • Reference-free preference optimization: Trains directly on (chosen, rejected) pairs by optimizing a policy log-probability difference with a target margin.
  • Efficient, DPO-style alternative: Provides a simpler alignment workflow that avoids reference-model dependencies and can achieve stronger preference outcomes.
  • Configurable loss and regularization: Supports sigmoid or hinge loss types, beta/margin tuning (beta and gamma-beta ratio), and optional SFT regularization to reduce capability forgetting.
  • Use cases: Preference alignment for instruction/chat models using UltraFeedback-style datasets, domain-specific alignment (e.g., math reasoning preferences), and controlled trade-offs between alignment strength and retention of base capabilities.

Quick Start

Use the simpo-training skill to train a Mistral 7B model on chosen/rejected preference pairs with a reference-free SimPO run configuration.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How does reference-free preference optimization work for LLM alignment?▼

Reference-free preference optimization trains a policy model directly on chosen and rejected response pairs by optimizing log-probability differences with a target margin, eliminating the need to maintain a separate reference model during alignment.

What is the best way to align a Mistral model without training a reference model?▼

The best way to align a Mistral model without a reference model is using SimPO, which applies a target margin to log-probability differences between chosen and rejected pairs, avoiding reference-model dependencies while achieving strong preference outcomes.

Can I use SFT regularization to prevent capability forgetting during SimPO training?▼

Yes, you can apply optional SFT regularization during SimPO training to reduce capability forgetting. This allows controlled trade-offs between preference alignment strength and retention of the base model's capabilities.

Does SimPO support hinge loss for preference optimization on PyTorch?▼

Yes, SimPO supports both sigmoid and hinge loss types for preference optimization on PyTorch. You can configure these loss functions alongside beta and gamma-beta ratio hyperparameters to tune the target margin.

Why use SimPO instead of DPO for instruction following alignment?▼

Use SimPO instead of DPO for instruction following alignment because it provides a simpler, reference-free workflow that avoids reference-model dependencies while achieving stronger preference outcomes on UltraFeedback-style datasets.

What dataset format is required for reference-free LLM alignment?▼

Reference-free LLM alignment requires preference datasets containing prompt, chosen, and rejected response triples. This format supports instruction following, helpfulness, and domain-specific tasks like math reasoning alignment.

Related Skills

Explore More Education & Research Skills→
ricablericable

fine-tuning-with-trl

Fine-tune large language models with TRL for supervised and preference alignment.

Community
Advanced
rocklambrosrocklambros
2

auditing-rlhf-reward-hacking

Audit RLHF, DPO, and RLAIF checkpoints for reward hacking by comparing reward win-rates with preference win-rates.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

simpo-training

Train LLMs with reference-free SimPO preference optimization using accelerate.

Community
Intermediate
gagan114662gagan114662

simpo-training

Train and align LLMs using reference-free Simple Preference Optimization.

Community
Advanced