skills.rest
AnandaAnugrahHandyanto
AnandaAnugrahHandyantoCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Optimize model preferences using beta-weighted SimPO loss without a reference model.

Updated Jun 19, 2026
Search Tags:#llm#alignment#training#simpo#dataset-prep#preference-optimization#loss-functions
One-click install
npx skills add https://github.com/AnandaAnugrahHandyanto/savarez_agent --skill simpo-training-anandaanugrahhandyanto
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/AnandaAnugrahHandyanto/savarez_agent/tree/main/optional-skills/mlops/simpo
Command: npx skills add https://github.com/AnandaAnugrahHandyanto/savarez_agent --skill simpo-training-anandaanugrahhandyanto

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

SimPO provides a reference-free approach to training alignment-focused language models by learning from preference data without a ground-truth reference model, enabling efficient and scalable policy optimization.

Core Features & Use Cases

  • Reference-free preference optimization for LLM alignment with improved sample efficiency.
  • Supports configurable hyperparameters (beta, gamma_beta_ratio, loss_type) and SFT regularization to preserve capabilities.
  • Use cases include fine-tuning assistants, safety-aligned chatbots, and instruction-following models using curated preference datasets.

Quick Start

Run the SimPO training script with a chosen base model and a prepared preference dataset to start alignment training.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I align an LLM without a reference model?▼

You can preserve base model capabilities during preference optimization by applying SFT regularization with a configurable sft_weight, preventing catastrophic forgetting while aligning the model's responses.

What hyperparameters do I tune for SimPO loss functions?▼

Key hyperparameters for SimPO loss functions include beta, gamma_beta_ratio, loss_type, and sft_weight, which control the optimization scale, target margin, loss formulation, and SFT preservation respectively.

How do I prepare datasets for reference-free preference optimization?▼

Dataset preparation for reference-free preference optimization requires curated preference datasets containing chosen and rejected response pairs, which are then processed end-to-end within the alignment training workflow.

Does reference-free preference optimization work for safety-aligned chatbots?▼

Reference-free preference optimization supports safety-aligned chatbots and instruction-following models by fine-tuning assistants directly on curated preference data without needing a pre-existing aligned reference model.

Related Skills

Explore More Data & Analytics Skills→
JamesFincherJamesFincher

simpo-training

Train LLMs with reference-free SimPO preference optimization from chosen and rejected pairs.

Community
Advanced
jinzhezenggroupjinzhezenggroup
124

mace

Configure ASE MACE backend parameters for ASE workflows.

Official
Intermediate
ovachieverovachiever
52

simpo-training

Optimize LLM preference alignment without a reference model using comparison data.

Community
Advanced
zhuangbiaoweizhuangbiaowei
2

simpo-training

Optimize LLM alignment with reference-free SimPO preference optimization.

Community
Advanced