fine-tuning-with-trl

Fine-tune and align language models using TRL's SFT, DPO, PPO, and GRPO trainers.

Updated May 13, 2026

Search Tags:#fine-tuning #dpo #trl #rlhf #grpo #llm alignment #reward modeling

Ready to run

npx skills add https://github.com/superfhp/lumi-agent-body --skill fine-tuning-with-trl-superfhp

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: fine-tuning-with-trl
Source: https://github.com/superfhp/lumi-agent-body/tree/main/skills/mlops/training/trl-fine-tuning
Command: npx skills add https://github.com/superfhp/lumi-agent-body --skill fine-tuning-with-trl-superfhp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trl, transformers, datasets, peft, accelerate, torch, and includes references (resource) components.

What problem does it solve? Aligning language models with human preferences requires complex post-training pipelines involving supervised fine-tuning, reward modeling, and reinforcement learning, which are difficult to implement correctly from scratch. ## Core Features & Use Cases - Full RLHF Pipeline: Run the complete SFT → Reward Model → PPO workflow to align a base model with human preferences. - Preference Alignment with DPO: Train on chosen/rejected pairs without a reward model, with 10+ loss variants including IPO, hinge, and robust DPO. - Memory-Efficient Online RL with GRPO: Train with custom reward functions for verifiable tasks like math and structured output, with LoRA support and a production-ready template. - Use Case: Fine-tune Qwen2.5-1.5B-Instruct on GSM8K with GRPO using correctness and format reward functions to produce structured reasoning answers. ## Quick Start Use the TRL fine-tuning skill to train Qwen2.5-0.5B with DPO on the ultrafeedback_binarized preference dataset.

Frequently Asked Questions about fine-tuning-with-trl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I fine-tune an LLM with DPO using TRL?▼

Load a preference dataset with prompt, chosen, and rejected fields, configure DPOConfig with beta and learning rate, then pass the model, tokenizer, and dataset to DPOTrainer and call train(). A CLI alternative is available via the trl dpo command.

What is the difference between SFT, DPO, PPO, and GRPO in TRL?▼

SFT trains on prompt-completion pairs for instruction following. DPO aligns with preference pairs without a reward model. PPO uses a trained reward model for maximum control. GRPO runs memory-efficient online RL by comparing groups of generated completions.

How much GPU memory does TRL training require?▼

For a 7B model, SFT needs about 16GB with LoRA, DPO about 24GB since it stores a reference model, PPO about 40GB for policy plus reward model, and GRPO about 24GB. LoRA, gradient checkpointing, and gradient accumulation reduce memory further.

Why does my GRPO training loss increase instead of decrease?▼

Increasing loss during GRPO training is expected behavior because the loss measures KL divergence from the initial policy as the model learns. Monitor reward, reward_std, and kl metrics instead of loss to track actual training progress.

How do I fix out-of-memory errors during DPO training?▼

Reduce per_device_train_batch_size to 1, lower max_length from 1024 to 512, and increase gradient_accumulation_steps to maintain the effective batch size. Enabling gradient checkpointing on the model also reduces memory usage.

When should I use Axolotl or Unsloth instead of TRL?▼

Use HuggingFace Trainer for basic fine-tuning without RL, Axolotl for YAML-based training configuration, LitGPT for minimal educational fine-tuning, and Unsloth for fast LoRA training. TRL is the right choice when you need RLHF, preference alignment, or reward modeling.