skills.rest
SethyPagna
SethyPagnaCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train language models with SimPO preference alignment without a reference model.

Updated May 14, 2026
Search Tags:#fine-tuning#huggingface#preference optimization#simpo#accelerate#llm alignment
One-click install
npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill simpo-training-sethypagna
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/SethyPagna/Secretary-Jarvis/tree/main/src/capabilities/optional-skills/mlops/simpo
Command: npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill simpo-training-sethypagna

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

SimPO training helps you align a language model to human preferences without the extra cost and complexity of a reference model, making preference optimization faster and easier to run.

Core Features & Use Cases

  • Reference-free alignment: Train directly on chosen/rejected response pairs instead of managing a separate baseline model.
  • Practical tuning guidance: Choose learning rates, beta, gamma-beta ratio, and SFT weight with documented recommendations for different model sizes and task types.
  • Dataset and loss selection: Prepare preference datasets, mix domains, and choose between sigmoid or hinge loss for general chat, reasoning, or instruction-following work.
  • Use case: Fine-tune a Mistral or Llama model on UltraFeedback-style preference data to improve helpfulness, reasoning quality, or domain-specific response quality.

Quick Start

Use this skill to train SimPO on your preferred preference dataset by selecting a model, setting beta, gamma_beta_ratio, and learning_rate, and launching the run through your alignment-handbook or Accelerate workflow.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I align an LLM with human preferences without using a reference model?▼

SimPO preference optimization aligns large language models without a reference model by training directly on chosen and rejected response pairs. This removes the need to manage a separate baseline model, making LLM alignment faster and less resource-intensive.

What format does preference data need to be in for SimPO training?▼

Preference data for SimPO training requires datasets containing chosen and rejected response pairs. You can mix domains and prepare these pairs for general chat, reasoning, or instruction-following tasks to optimize the model effectively.

How do I configure beta and learning rate for preference optimization on Mistral or Llama models?▼

Configure preference optimization by setting beta, the gamma-beta ratio, and learning rate. The process provides documented recommendations for different model sizes and task types, allowing you to apply SFT regularization and choose between sigmoid or hinge loss.

Does SimPO training work with Hugging Face Accelerate for fine-tuning instruct models?▼

Yes, SimPO training works with Hugging Face Accelerate. You can launch your training run through an alignment-handbook or Accelerate workflow to fine-tune both base and instruct models on UltraFeedback-style preference data.

When should I choose sigmoid vs hinge loss for reference-free preference optimization?▼

Choose sigmoid or hinge loss based on your specific task type, such as general chat, reasoning, or domain-specific instruction following. The loss function selection works alongside configurable beta, gamma-beta ratio, and SFT weight to optimize alignment quality.

Why use SimPO instead of standard preference optimization methods that require a baseline?▼

Use SimPO to avoid the extra cost and complexity of managing a reference model during preference optimization. By training directly on chosen and rejected response pairs, it achieves faster alignment for base and instruct models without baseline overhead.

Related Skills

Explore More Education & Research Skills→
Supporter09Supporter09

fine-tuning-with-trl

Fine-tune Hugging Face Transformers models with TRL for instruction alignment and preference optimization.

Community
Advanced
ScientiaCapitalScientiaCapital
1

model-deployment

Export fine-tuned models to GGUF and deploy with vLLM or Ollama.

Community
Advanced
jacardljacardl

simpo-training

Optimize language model alignment without a reference model using preference datasets.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

simpo-training

Train LLMs with reference-free SimPO preference optimization using accelerate.

Community
Intermediate