skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Optimize LLMs with reference-free preference alignment using sigmoid and hinge losses.

2|Updated Apr 12, 2026
Search Tags:#simpo#deep-learning#post-training#preference-optimization#llm-alignment#dpo-alternative
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill simpo-training-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/06-post-training/simpo
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill simpo-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

SimPO provides a simple, efficient way to align large language models to human preferences using chosen/rejected pairs without requiring a separate reference model, reducing compute and implementation complexity compared to DPO or PPO.

Core Features & Use Cases

  • Reference-free preference optimization: Directly optimizes the policy using chosen vs rejected responses, removing the need for a reference model.
  • Configurable losses and hyperparameters: Supports sigmoid and hinge losses, tunable beta and gamma_beta_ratio, and optional SFT regularization to preserve capabilities.
  • Practical workflows and troubleshooting: Includes recommended configs for Mistral 7B and Llama 3 8B, dataset preparation guidance, and common fixes for divergence, OOM, and capability loss.
  • Use Case: Fine-tune a 7B instruction model on UltraFeedback-style preference pairs to improve helpfulness and alignment while keeping training single-node and resource-efficient.

Quick Start

Run the provided SimPO training pipeline with your preference dataset and an accelerate configuration to fine-tune your LLM using the recommended SimPO hyperparameters.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I align an LLM using preference data without training a reference model?▼

You can align an LLM without a reference model by using SimPO, a reference-free preference optimization method that directly optimizes the policy on chosen and rejected response pairs, reducing compute complexity compared to DPO.

Can I fine-tune a 7B instruction model on UltraFeedback preference pairs using a single-node setup?▼

Yes, you can fine-tune a 7B instruction model on UltraFeedback-style preference pairs using SimPO, which integrates with DeepSpeed and accelerate to keep post-training resource-efficient on a single node.

What are the best hyperparameters for Llama 3 8B preference optimization?▼

For Llama 3 8B preference optimization, use SimPO's recommended configurations, which include tunable beta and gamma_beta_ratio, optional SFT regularization, and support for sigmoid and hinge loss types to prevent capability loss.

How does reference-free preference optimization compare to DPO for LLM post-training?▼

Unlike DPO, reference-free preference optimization like SimPO eliminates the need for a separate reference model, which significantly reduces implementation complexity and compute requirements while directly optimizing the policy using preference data.

Why does my preference optimization training diverge or run out of memory?▼

Training divergence or out-of-memory errors during preference optimization can often be resolved by applying SimPO's common fixes, which include adjusting configurable hyperparameters and utilizing DeepSpeed and accelerate for efficient memory management.

Do I need PyTorch and Hugging Face datasets to run SimPO for LLM alignment?▼

Yes, SimPO integrates with PyTorch, Hugging Face datasets, and TRL, requiring these frameworks to execute the preference optimization pipeline and load chosen/rejected pairs for large language model alignment.

Related Skills

Explore More Education & Research Skills→
JamesFincherJamesFincher

simpo-training

Train LLMs with reference-free SimPO preference optimization from chosen and rejected pairs.

Community
Advanced
DaddyElonMusk69DaddyElonMusk69

simpo-training

Trains LLMs with reference-free SimPO optimization using labeled preference datasets and tunable hyperparameters.

Community
Advanced
eyadsibaieyadsibai
7

llm-training

Compare frameworks and techniques for training and finetuning large language models.

Community
Intermediate
nirukk52nirukk52
1

e2e-testing

Automate ScreenGraph end-to-end UI validation with Playwright and .env configuration.

Community
Intermediate