skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

trl-fine-tuning

Train and align language models using TRL SFT, DPO, GRPO, and RLOO methods.

Updated Sep 10, 2026
Search Tags:#fine-tuning#TRL#RLHF#GRPO#dpo#llm alignment#reward model
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill trl-fine-tuning-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trl-fine-tuning
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/training/trl-fine-tuning
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill trl-fine-tuning-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trl, transformers, datasets, peft, accelerate, torch, and includes references (resource) components.

What problem does it solve? Aligning language models with human preferences requires navigating multiple post-training methods, dataset formats, and hyperparameter choices. This Skill provides complete, working workflows for supervised fine-tuning, preference alignment, reward modeling, and online reinforcement learning with HuggingFace TRL. ## Core Features & Use Cases - Full RLHF Pipeline: Step-by-step checklists for SFT, reward model training, and RLOO reinforcement learning, with both Python and CLI examples. - Preference Alignment with DPO: Train on chosen/rejected pairs without a reward model, including 10+ loss variants (IPO, hinge, robust, APO) with recommended hyperparameters. - Memory-Efficient GRPO: Deep guidance on reward function design, mode collapse detection, multi-stage training, and a production-ready training template. - Use Case: Fine-tune Qwen2.5-1.5B-Instruct on GSM8K math problems using GRPO with correctness and format reward functions, LoRA adapters, and 8-bit optimization on a single GPU. ## Quick Start Ask the agent to fine-tune a Qwen model with DPO on a preference dataset using TRL and save the aligned model.

Frequently Asked Questions about trl-fine-tuning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fine-tune an LLM with DPO using TRL?▼

Load a preference dataset with prompt, chosen, and rejected fields, configure DPOConfig with beta and learning rate, then train with DPOTrainer. You can also run the CLI command trl dpo with your model and dataset names.

What is the difference between DPO, GRPO, and RLOO in TRL?▼

DPO aligns models directly on preference pairs without a reward model. RLOO uses a trained reward model for online RL, replacing the removed PPO trainer. GRPO compares multiple generations per prompt with reward functions and uses less memory.

Was PPO removed from TRL and what replaces it?▼

Yes, PPOTrainer and PPOConfig were removed in TRL 1.x. Use RLOOTrainer or the trl rloo CLI for reward-model-driven RLHF, or GRPOTrainer for memory-efficient online RL with reward functions.

How much GPU memory does TRL training require?▼

For 7B models, SFT needs about 16GB with LoRA, DPO about 24GB since it stores a reference model, RLOO about 40GB for policy plus reward model, and GRPO about 24GB. Use LoRA, gradient checkpointing, and gradient accumulation to reduce memory.

Why does GRPO loss increase during training?▼

Rising GRPO loss is expected because it measures KL divergence from the initial policy as the model learns. Monitor reward metrics and reward_std instead; reward should increase while reward_std stays above 0.1 to avoid mode collapse.

How do I fix OOM errors during DPO training?▼

Reduce per_device_train_batch_size to 1, lower max_length to 512, and increase gradient_accumulation_steps to maintain effective batch size. Enabling gradient checkpointing on the model also reduces memory usage.

Related Skills

Explore More Software Engineering Skills→
thinking-machines-labthinking-machines-lab
4.0k

dpo

Configure and run Direct Preference Optimization training with a model and preference dataset.

Official
Intermediate
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
NVIDIANVIDIA
49

sop-ddm-finetuning

Fine-tune DDM-Net temporal boundary detectors on annotated SOP video datasets.

Official
Advanced
xu1713xu1713

unsloth

Fine-tune LLMs with LoRA and QLoRA using Unsloth's optimized training framework.

Community
Intermediate