skills.rest
Jensen-Yao
Jensen-YaoCOMMUNITY
Explore all skills by this author →
GitHub

verl-rl-training

Configures and launches reinforcement learning training for LLMs using verl with PPO, GRPO, and distributed backends.

2|Updated Aug 15, 2026
Search Tags:#reinforcement learning#LLM training#RLHF#distributed training#PPO#GRPO#verl
One-click install
npx skills add https://github.com/Jensen-Yao/agents-skills --skill verl-rl-training-jensen-yao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/Jensen-Yao/agents-skills/tree/main/skills/verl
Command: npx skills add https://github.com/Jensen-Yao/agents-skills --skill verl-rl-training-jensen-yao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires verl, torch, ray, vllm, transformers, and includes references (resource) components.

What problem does it solve? Training large language models with reinforcement learning (RLHF, GRPO, PPO) requires coordinating distributed training backends, rollout engines, reward functions, and cluster resources, which is complex and error-prone to configure from scratch. ## Core Features & Use Cases - Multi-Algorithm RL Training: Supports PPO, GRPO, DAPO, RLOO, ReMax, and REINFORCE++ with configurable advantage estimators and KL control. - Flexible Backend Swapping: Mix FSDP, FSDP2, or Megatron-LM training backends with vLLM, SGLang, or HuggingFace rollout engines for models from 0.5B to 671B parameters. - Troubleshooting Guidance: Diagnoses OOM errors, training instability, slow weight synchronization, and vLLM version mismatches with concrete configuration fixes. - Use Case: Train a math reasoning model on GSM8K with GRPO by preparing a parquet dataset, defining a rule-based reward function, and launching an 8-GPU training run with vLLM rollouts. ## Quick Start Use the verl skill to set up a GRPO training run for Qwen2.5-7B on my GSM8K parquet dataset with 8 GPUs and vLLM rollouts.

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with GRPO using verl?▼

Prepare a parquet dataset with prompt and reward_model columns, then run python3 -m verl.trainer.main_ppo with algorithm.adv_estimator=grpo and rollout.n set above 1. Enable use_kl_loss with a small kl_loss_coef like 0.001 for stable GRPO training.

What is the difference between GRPO and PPO in verl?▼

GRPO is critic-free and estimates advantages from grouped samples, making it suited for sparse-reward tasks like math reasoning. PPO uses adv_estimator=gae with a separate critic model and works better for dense reward signals.

Does verl support Megatron-LM for large model training?▼

Yes, verl supports Megatron-LM as a training backend for models above 70B parameters with tensor and pipeline parallelism. Install the mbridge package for checkpoint format conversion, and note Megatron is not supported on AMD GPUs.

Why does verl run out of memory during rollout?▼

Rollout OOM happens when generation batches exceed GPU memory. Reduce log_prob_micro_batch_size, enable gradient checkpointing, lower vLLM gpu_memory_utilization to 0.7, or switch to FSDP2 with CPU offloading.

Which vLLM versions are compatible with verl?▼

verl requires vLLM versions between 0.8.2 and 0.12.0, and vLLM 0.7.x should be avoided due to known bugs. For vLLM 0.8.x issues, set enforce_eager to true to disable CUDA graphs.

When should I use TRL instead of verl for LLM training?▼

Use TRL or Axolotl when you only need simple supervised fine-tuning or DPO without distributed RL infrastructure. verl is designed for production-scale RL training with multi-node clusters and flexible backend swapping.

Related Skills

Explore More Software Engineering Skills→
thinking-machines-labthinking-machines-lab
4.0k

tinker-cli

Automate Tinker training runs and checkpoint management via CLI, including HuggingFace Hub publishing.

Official
Intermediate
kamelmhkamelmh

fine-tuning-expert

Prepare and tune large language models with LoRA, QLoRA, and PEFT.

Community
Advanced
qcmuuqcmuu

openrlhf-training

Run distributed OpenRLHF RLHF training on Ray with vLLM acceleration.

Community
Advanced
helix4uhelix4u

slime-rl-training

Guide RL-based post-training of LLMs with Megatron-LM and SGLang.

Community
Advanced