simpo-training

Reference-free LLM preference learning optimizes alignment directly from a preference dataset without a dedicated reference policy. SimPO applies reference-free policy optimization to accelerate experimentation across various model sizes.

Reference-free preference optimization is a technique that aligns base models to user preferences without loading a separate reference policy. It supports instruction fine-tuning, safety alignment, and user-preference learning using a small dataset.

Configure SimPO training by setting exposed hyperparameters like beta, gamma_beta_ratio, loss_type, sft_weight, and dataset_mixer. The workflow integrates natively with PyTorch, Transformers, Datasets, TRL, and Accelerate for execution.

Yes, reference-free preference optimization supports safety alignment and instruct-model fine-tuning across various model sizes. It enables efficient, model-agnostic alignment by eliminating the need for a dedicated reference policy.

No, a dedicated reference policy is not required for preference learning. This approach eliminates the reference model dependency entirely, enabling faster experimentation and model-agnostic alignment using your preference dataset.

The best way to start reference-free LLM alignment is to prepare a base model and a preference dataset. You then run the SimPO training workflow with your chosen hyperparameter configuration to optimize preferences.