skills.rest
t2ance
t2anceCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Optimize LLM alignment with reference-free SimPO preference optimization.

Updated Aug 27, 2026
Search Tags:#llm#efficiency#alignment#simpo#preference#post-training
One-click install
npx skills add https://github.com/t2ance/dr-claw-plugin --skill simpo-training-t2ance
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/t2ance/dr-claw-plugin/tree/main/plugins/ml-training-stack/skills/post-training/simpo
Command: npx skills add https://github.com/t2ance/dr-claw-plugin --skill simpo-training-t2ance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve?

SimPO provides a reference-free method for optimizing user preferences in LLM alignment, reducing reliance on a separate reference model and enabling streamlined post-training refinement.

Core Features & Use Cases

  • Direct, reference-free preference optimization for faster alignment with minimal overhead.
  • Supports configurable hyperparameters (learning_rate, beta, gamma_beta_ratio, loss_type, sft_weight) and flexible dataset integration for diverse tasks.
  • Use cases include post-training instruct/fine-tuning, preference data experimentation, and lightweight alignment experiments.

Quick Start

Run a small simpo training job with your chosen model and ultrafeedback dataset to begin alignment.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM alignment without a reference model?▼

SimPO enables reference-free preference optimization by directly optimizing preferred outputs over rejected ones. It eliminates the need for a separate reference model during post-training, reducing overhead while enabling efficient and scalable fine-tuning across domains.

What's the best way to fine-tune an LLM using preference data?▼

Apply a reference-free preference optimization approach to fine-tune your LLM using preference data. This method directly compares preferred and rejected outputs, requiring configuration of learning_rate, beta, gamma_beta_ratio, loss_type, and optional sft_weight to streamline post-training refinement.

Does SimPO preference optimization work with Hugging Face transformers and TRL?▼

SimPO preference optimization integrates with torch, transformers, datasets, and trl. This stack supports flexible dataset integration and configurable hyperparameters for diverse alignment tasks and lightweight experiments within the Hugging Face ecosystem.

Why use reference-free preference optimization instead of standard RLHF?▼

Reference-free preference optimization reduces reliance on a separate reference model, enabling faster alignment with minimal overhead. It directly optimizes preferred outputs over rejected ones, streamlining post-training refinement and making scalable fine-tuning more efficient.

How do I configure hyperparameters for SimPO post-training?▼

Configure learning_rate, beta, gamma_beta_ratio, loss_type, and optional sft_weight for SimPO post-training. These hyperparameters control the reference-free preference optimization process, allowing you to balance alignment objectives and fine-tuning efficiency across diverse domains.

Related Skills

Explore More Data & Analytics Skills→
OpenLAIROpenLAIR
1.0k

simpo-training

Run SimPO training to optimize LLM preferences without a reference model.

Official
Intermediate
ovachieverovachiever
52

simpo-training

Optimize LLM preference alignment without a reference model using comparison data.

Community
Advanced
xiaoquqixiaoquqi

fine-tuning-with-trl

Implement RLHF fine-tuning workflows with TRL for instruction and preference alignment.

Community
Advanced
hochoa13hochoa13

fine-tuning-with-trl

Fine-tune LLMs with SFT, DPO, PPO/GRPO, and reward model training.

Community
Advanced