verl-rl-training

Train large language models with GRPO and PPO using verl.

11.5k|842|Updated Nov 3, 2025

Search Tags:#reinforcement learning #distributed training #ppo #rlhf #grpo #post-training #llm training

Ready to run

npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill verl-rl-training

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/06-post-training/verl
Command: npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill verl-rl-training

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the complexity and resource demands of training Large Language Models (LLMs) using Reinforcement Learning (RL) at scale, providing a robust framework for advanced post-training techniques.

Core Features & Use Cases

Scalable RL Training: Supports training up to 671B parameter models using distributed backends like FSDP and Megatron-LM.
Flexible RL Algorithms: Implements PPO, GRPO, RLOO, REINFORCE++, and more, with support for custom reward functions.
Backend Agnosticism: Seamlessly switch between rollout engines (vLLM, SGLang) and training backends (FSDP, Megatron-LM).
Use Case: Fine-tune a Llama-3 model using GRPO on a math reasoning dataset to improve its problem-solving capabilities, leveraging a multi-GPU cluster for efficient training.

Quick Start

Use the verl-rl-training skill to launch a GRPO training job for a Qwen2.5-7B model on the GSM8K dataset using 8 GPUs.

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scale reinforcement learning training for large language models?▼

Reinforcement learning training for LLMs supports advanced algorithms like GRPO and PPO, enabling multi-turn interactions and complex agentic workflows by utilizing flexible backend integrations such as FSDP and Megatron-LM.

Can I use PPO and GRPO for LLM post-training with custom reward functions?▼

Yes, LLM post-training supports flexible RL algorithms including PPO, GRPO, RLOO, and REINFORCE++, allowing you to implement custom reward functions for fine-tuning models on specific reasoning datasets.

How do I start a GRPO training job on a multi-GPU cluster?▼

To start GRPO training on a multi-GPU cluster, configure your rollout engine and training backend, then launch the job targeting your chosen model, such as a Qwen2.5-7B, using 8 GPUs for distributed processing.

Does vLLM work with FSDP for distributed RL training?▼

Yes, vLLM works with FSDP for distributed RL training through backend agnosticism, allowing you to seamlessly switch between rollout engines like vLLM and SGLang and training backends like FSDP and Megatron-LM.

What is the best way to train vision-language models with reinforcement learning?▼

The best way to train vision-language models with reinforcement learning is using a framework that supports multimodal research and complex agentic workflows, enabling multi-turn interactions alongside scalable distributed training.