dpo

To set up Direct Preference Optimization training, provide your chosen model name, custom preference dataset, learning rate, and DPO beta parameters. The workflow processes chosen and rejected preference pairs to configure and initialize the training run.

Yes, you can start DPO training from a base or SFT checkpoint by providing the optional load_checkpoint_path parameter. This initializes the Direct Preference Optimization run using your previously fine-tuned model weights.

Built-in preference datasets supported for DPO fine-tuning include HHH, HelpSteer3, and UltraFeedback. You can select any of these datasets alongside your model name to quickly configure and run Direct Preference Optimization training.

Required parameters to configure a DPO training run include model_name, dataset, learning_rate, lr_schedule, dpo_beta, and reference_model_name. You can also supply load_checkpoint_path to initialize training from a previous checkpoint.

DPO training with the Tinker API configures Direct Preference Optimization by applying chosen and rejected comparison pairs to a specified model. It refines model alignment using human preference data from built-in or custom datasets.

Use DPO training instead of standard supervised fine-tuning when you have human preference data containing chosen and rejected response pairs. Direct Preference Optimization refines model alignment based on these comparisons rather than simple input-output mappings.