fine-tuning-with-trl

Fine-tune LLMs with SFT, DPO, PPO, and GRPO using HuggingFace Transformers.

2|Updated Feb 4, 2026

Search Tags:#llm #fine-tuning #reinforcement learning #dpo #trl #sft #preference alignment

Ready to run

npx skills add https://github.com/zhuangbiaowei/smart_bot --skill fine-tuning-with-trl-zhuangbiaowei

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: fine-tuning-with-trl
Source: https://github.com/zhuangbiaowei/smart_bot/tree/main/skills/trl-fine-tuning
Command: npx skills add https://github.com/zhuangbiaowei/smart_bot --skill fine-tuning-with-trl-zhuangbiaowei

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trl, transformers, datasets, peft, accelerate, torch, and includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of aligning Large Language Models (LLMs) with human preferences and instructions, making them more helpful, harmless, and honest.

Core Features & Use Cases

Supervised Fine-Tuning (SFT): Instruction tuning for better response generation.
Direct Preference Optimization (DPO): Aligning models with human preferences without a separate reward model.
Reinforcement Learning (PPO/GRPO): Optimizing models using reward signals for advanced alignment.
Reward Model Training: Creating models that score the quality of LLM outputs.
Use Case: You have a base LLM and want to fine-tune it to follow user instructions more accurately or to adopt a specific persona. You can use SFT for initial instruction tuning and then DPO or PPO with preference data to further refine its behavior.

Quick Start

Use the fine-tuning-with-trl skill to perform supervised fine-tuning on the 'Qwen/Qwen2.5-0.5B' model using the 'trl-lib/Capybara' dataset.

Frequently Asked Questions about fine-tuning-with-trl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I align an LLM with human preferences using reinforcement learning?▼

Align an LLM with human preferences using reinforcement learning by applying SFT, DPO, PPO, or GRPO techniques. This Skill facilitates preference alignment and reward model training for HuggingFace Transformers architectures.

What is the best way to fine-tune a HuggingFace Transformer for instruction following?▼

The best way to fine-tune a HuggingFace Transformer for instruction following is Supervised Fine-Tuning (SFT). You can use this Skill to instruction tune a base LLM to follow user instructions more accurately.

Can I use DPO to align models without training a separate reward model?▼

Yes, you can use Direct Preference Optimization (DPO) to align models without a separate reward model. DPO directly optimizes the model against preference data, bypassing the need for explicit reward model training.

Do I need torch and peft to perform PPO and GRPO on large language models?▼

Yes, you need torch, peft, transformers, datasets, and accelerate to perform PPO and GRPO. These dependencies provide the necessary environment for optimizing models using reward signals for advanced alignment.

When should I use SFT before applying PPO for preference alignment?▼

You should use SFT before applying PPO for initial instruction tuning. SFT adapts the base LLM to follow instructions, which is necessary before refining its behavior with preference data using PPO or DPO.