skills.rest
hung-phan
hung-phanCOMMUNITY
Explore all skills by this author →
GitHub

unsloth-advanced

Train unsloth models with GRPO, DPO, ORPO, CPT, and vision fine-tuning.

1|Updated Jun 4, 2026
Search Tags:#reasoning#vision#unsloth#preference alignment#continued pretraining
One-click install
npx skills add https://github.com/hung-phan/ml-skills --skill unsloth-advanced
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: unsloth-advanced
Source: https://github.com/hung-phan/ml-skills/tree/main/skills/ml-review/references/ml-training/unsloth-advanced
Command: npx skills add https://github.com/hung-phan/ml-skills --skill unsloth-advanced

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires unsloth, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the need for advanced techniques in model training, enabling deeper reasoning, preference alignment, and vision capabilities for unsloth models.

Core Features & Use Cases

  • GRPO (Group Relative Policy Optimization): Enhances reasoning using custom Python reward functions without a value model, suitable for multi-GPU scenarios.
  • DPO (Direct Preference Optimization): Aligns models with human preferences using paired preference data without a reward model.
  • ORPO (Odds Ratio Preference Optimization): Combines SFT and preference alignment in a single training pass.
  • Continued Pretraining (CPT): Teaches new knowledge or languages to models.
  • Vision Model Fine-Tuning: Fine-tunes vision-language models for image understanding.
  • Use Case: When building reasoning models, aligning with preferences, fine-tuning vision-language models, or teaching new knowledge/languages.

Quick Start

Train a GRPO model using the provided script and data.

Frequently Asked Questions about unsloth-advanced

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I enhance reasoning in unsloth models without using a value model?▼

You can enhance reasoning using Group Relative Policy Optimization (GRPO), which relies on custom Python reward functions instead of a value model. This approach is particularly suitable for multi-GPU training scenarios.

What is the difference between DPO and ORPO for preference alignment?▼

Direct Preference Optimization (DPO) aligns models using paired preference data without a reward model, while Odds Ratio Preference Optimization (ORPO) combines supervised fine-tuning and preference alignment in a single training pass.

Can I fine-tune vision-language models for image understanding with unsloth?▼

Yes, you can fine-tune vision-language models for image understanding. This process requires providing vision datasets specifically formatted for training the model on visual tasks.

How do I teach new knowledge or languages to an existing model?▼

You can teach new knowledge or languages to a model through Continued Pretraining (CPT). This technique allows you to inject fresh domain knowledge or linguistic capabilities into your existing architecture.

What data format is required for Direct Preference Optimization?▼

Direct Preference Optimization requires paired preference data. You must provide datasets containing comparisons of preferred and rejected responses to align the model with human preferences without a separate reward model.

Related Skills

Explore More Software Engineering Skills→
ZardLi1115ZardLi1115
4

fine-tuning-with-trl

Align LLMs with TRL trainers for SFT, DPO, PPO, GRPO, and reward modeling.

Community
Intermediate
JKhyroJKhyro

axolotl

Configure and validate Axolotl YAML files for model fine-tuning workflows.

Community
Advanced
JKhyroJKhyro

fine-tuning-with-trl

Fine-tune language models with TRL for SFT, DPO, PPO, and GRPO training.

Community
Advanced
DifanaDAPDifanaDAP

fine-tuning-with-trl

Train and align language models with human preferences using TRL techniques.

Community
Advanced