simpo-training

Optimize LLM preferences with reference-free SimPO training.

Updated Apr 30, 2026

Search Tags:#LLM #alignment #preference optimization #SimPO #alignment algorithm

Ready to run

npx skills add https://github.com/Ced3-han/Harness-Settings --skill simpo-training-ced3-han

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/Ced3-han/Harness-Settings/tree/main/skills/simpo
Command: npx skills add https://github.com/Ced3-han/Harness-Settings --skill simpo-training-ced3-han

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of preference alignment in large language models (LLMs), offering a reference-free alternative to DPO (Diverse Preference Optimization) for more efficient and simpler training.

Core Features & Use Cases

Reference-Free: No need for a reference model, simplifying the alignment process.
Performance: Demonstrates better performance than DPO on AlpacaEval 2.0.
Efficiency: More efficient than DPO, suitable for limited compute resources.
Use Case: Ideal for preference alignment when seeking a simpler, faster training process compared to DPO/PPO.

Quick Start

Install the SimPO skill and train a model on preference data using the provided configuration files.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I optimize LLM preferences without a reference model using SimPO?▼

To optimize LLM preferences without a reference model, SimPO provides a reference-free alternative to DPO that simplifies alignment training. It eliminates the need for a separate reference model, accelerating the preference optimization process while maintaining performance.

What is the difference between SimPO and DPO for preference alignment?▼

The difference between SimPO and DPO is that SimPO is a reference-free alignment algorithm, whereas DPO requires a reference model. SimPO demonstrates better performance and efficiency than DPO on AlpacaEval 2.0, making it suitable for limited compute resources.

How do I train an LLM with preference data using transformers and trl?▼

To train an LLM with preference data using transformers and trl, you can use the provided SimPO scripts. The process requires torch, transformers, datasets, trl, and accelerate libraries to execute the preference alignment training on your dataset.

Do I need accelerate and datasets libraries to run SimPO alignment?▼

Yes, you need accelerate and datasets libraries to run SimPO alignment. The Skill also requires torch, transformers, and trl to properly execute the large language model preference optimization workflows.

Why use reference-free preference optimization for large language models?▼

You use reference-free preference optimization for large language models to simplify the alignment process and reduce memory overhead. By removing the reference model dependency, SimPO enables faster training and better efficiency compared to traditional DPO methods.

Can I use SimPO for preference alignment with limited compute resources?▼

Yes, you can use SimPO for preference alignment with limited compute resources. Because it is a reference-free method, it is more efficient than DPO and specifically designed to simplify and accelerate LLM training on constrained compute environments.