skills.rest
hhhi21g
hhhi21gCOMMUNITY
Explore all skills by this author →
GitHub

verl-rl-training

Train large language models with reinforcement learning on distributed infrastructure.

Updated Apr 11, 2026
Search Tags:#reinforcement learning#distributed training#large language models#transformers#ppo#vllm#grpo
One-click install
npx skills add https://github.com/hhhi21g/HealthCenter --skill verl-rl-training-hhhi21g
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/hhhi21g/HealthCenter/tree/main/.codex/skills/verl
Command: npx skills add https://github.com/hhhi21g/HealthCenter --skill verl-rl-training-hhhi21g

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of training large language models (LLMs) with reinforcement learning (RL) at scale, offering a flexible infrastructure backend for RLHF, GRPO, PPO, and other RL algorithms.

Core Features & Use Cases

  • Scalable RL Training: Supports RL training for LLMs up to 671B parameters.
  • Flexible Backends: Allows swapping backends like FSDP, Megatron-LM, and vLLM.
  • Multiple RL Algorithms: Includes support for PPO, GRPO, RLOO, REINFORCE++, and DAPO.
  • Multi-Turn Rollout: Enables multi-turn rollout with tool calling for agentic workflows.
  • Vision-Language Models: Supports vision-language model RL training.
  • Use Case: Ideal for researchers and engineers looking to implement distributed RL training for LLMs, with a focus on math reasoning and reasoning models.

Quick Start

Train a reasoning model on math tasks like GSM8K or MATH using verl with the following command:

python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo \
  data.train_files=~/data/gsm8k/train.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-7B \
  actor_rollout_ref.rollout.n=8 \
  actor_rollout_ref.actor.use_kl_loss=True \
  trainer.n_gpus_per_node=8

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large language models with reinforcement learning using PPO or GRPO?▼

Train large language models with PPO or GRPO by running distributed RL training environments that support flexible backends like FSDP, Megatron-LM, and vLLM. Use the main trainer module to execute algorithms such as GRPO on math reasoning datasets.

Can I use vLLM for rollout when training LLMs with reinforcement learning?▼

Yes, you can use vLLM for rollout when training LLMs with reinforcement learning. The distributed training infrastructure allows swapping backends like vLLM, FSDP, and Megatron-LM to optimize the rollout phase for models up to 671B parameters.

What is the best way to scale reinforcement learning training for 671B parameter models?▼

The best way to scale reinforcement learning training for 671B parameter models is using a distributed infrastructure that integrates Ray for orchestration and supports flexible backends like Megatron-LM and FSDP to handle large-scale LLM training efficiently.

Do I need Ray and vLLM to run distributed RL training for LLMs?▼

Yes, you need Ray and vLLM along with verl, torch, and transformers to run distributed RL training for LLMs. These dependencies provide the necessary orchestration, rollout generation, and model handling capabilities for scalable training.

Does reinforcement learning training support vision-language models and multi-turn rollout?▼

Yes, reinforcement learning training supports vision-language models and multi-turn rollout. The infrastructure enables multi-turn rollout with tool calling for agentic workflows and includes RL training capabilities for vision-language model architectures.

Why use GRPO instead of PPO for training reasoning models on math tasks?▼

Use GRPO instead of PPO for training reasoning models on math tasks to leverage algorithmic variations optimized for specific outcomes. The framework supports PPO, GRPO, RLOO, REINFORCE++, and DAPO, allowing flexible algorithm selection for math reasoning.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

openrlhf-training

Train large language models with RLHF using Ray and vLLM acceleration.

Community
Advanced
swaruplabswaruplab
94

torch-geometric

Build and benchmark Graph Neural Networks with PyTorch Geometric.

Community
Advanced
KappTech88KappTech88

verl-rl-training

Orchestrate PPO/GRPO RL training for large language models with verl.

Community
Advanced
arsityarsity

verl-rl-training

Train LLMs with reinforcement learning at scale using Verl.

Community
Advanced