simpo-training

Optimize language model alignment without a reference model using preference datasets.

Updated Apr 16, 2026

Search Tags:#mlops #llm #alignment #training #simpo #preference

Ready to run

npx skills add https://github.com/jacardl/New-Radar --skill simpo-training-jacardl

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/jacardl/New-Radar/tree/main/backend/frameworks/hermes-agent/optional-skills/mlops/simpo
Command: npx skills add https://github.com/jacardl/New-Radar --skill simpo-training-jacardl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve?

Simple Preference Optimization (SimPO) provides a reference-free approach to aligning large language models by learning from human preferences, eliminating the need for a reference model and improving training efficiency.

Core Features & Use Cases

Reference-free preference optimization for LLM alignment.
Configurable hyperparameters (beta, gamma, learning rate) and support for various model sizes.
Use cases include general instruction following, reasoning tasks, and domain-specific alignment experiments.

Quick Start

Run the provided training script with your model and a prepared preference dataset to begin training.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is reference-free preference optimization for LLM alignment?▼

Reference-free preference optimization aligns large language models using human-like preferences without requiring a separate reference model. This approach improves training efficiency by eliminating the reference model overhead while still learning from preference data.

How do I align a 7B to 70B language model using SimPO?▼

To align models from 7B to 70B using SimPO, run the provided training script with your chosen model and a prepared preference dataset. You configure hyperparameters like beta, gamma, and learning rate to drive the training process.

Do I need a reference model for preference optimization with TRL and Transformers?▼

No, you do not need a reference model for preference optimization using this approach. SimPO is a reference-free method that works with TRL and Transformers to align models directly from human-like preferences.

Can I use SimPO for domain-specific alignment and reasoning tasks?▼

Yes, you can use SimPO for domain-specific alignment experiments, general instruction following, and reasoning tasks. It supports configurable hyperparameters and various model sizes to accommodate different alignment scenarios.

What dependencies are required to run SimPO training?▼

SimPO training requires PyTorch, Transformers, Datasets, TRL, and Accelerate. You also need a prepared preference dataset and a SimPO configuration to properly drive the training process.