skills.rest
MedTiLab
MedTiLabCOMMUNITY
Explore all skills by this author →
GitHub

verl-rl-training

Implements reinforcement learning training for LLMs using verl with PPO, GRPO, and distributed backends.

5|2|Updated Aug 10, 2026
Search Tags:#reinforcement learning#RLHF#distributed training#PPO#GRPO#verl#llm post-training
One-click install
npx skills add https://github.com/MedTiLab/Auto-meta-analysis --skill verl-rl-training-medtilab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/MedTiLab/Auto-meta-analysis/tree/main/skills/post-training/verl
Command: npx skills add https://github.com/MedTiLab/Auto-meta-analysis --skill verl-rl-training-medtilab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires verl, torch, ray, vllm, transformers, and includes references (resource) components.

What problem does it solve? Training large language models with reinforcement learning (RLHF, GRPO, PPO) requires coordinating distributed training backends, rollout engines, reward functions, and weight synchronization, which is complex to configure and debug without structured guidance. ## Core Features & Use Cases - Multi-Algorithm RL Training: Configure and launch PPO, GRPO, DAPO, RLOO, REINFORCE++, and other algorithms with YAML-based configs for models from 0.5B to 671B parameters. - Flexible Backend Orchestration: Swap training backends (FSDP, FSDP2, Megatron-LM) and rollout engines (vLLM, SGLang, HuggingFace) via Ray-based HybridFlow architecture. - Troubleshooting & Scaling Guidance: Resolve OOM, training instability, weight sync, and multi-node NCCL issues, plus support for LoRA RL, vision-language models, and multi-turn tool calling. - Use Case: Train a math reasoning model on GSM8K with GRPO by preparing a parquet dataset, defining a rule-based reward function, and launching an 8-GPU training run with vLLM rollout. ## Quick Start Use the verl skill to set up a GRPO training run for Qwen2.5-7B on a GSM8K parquet dataset with vLLM rollout across 8 GPUs.

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with GRPO using verl?▼

Prepare a parquet dataset with prompt and reward_model columns, set algorithm.adv_estimator=grpo with rollout.n greater than 1, and launch via python3 -m verl.trainer.main_ppo. Enable use_kl_loss with a small kl_loss_coef such as 0.001 for stable GRPO training.

What is the difference between GRPO and PPO in verl?▼

GRPO is critic-free and estimates advantages from grouped samples, suiting sparse-reward tasks like math reasoning. PPO uses adv_estimator=gae with a separate critic model and Generalized Advantage Estimation, which works better for dense reward signals.

Does verl support Megatron-LM for large model training?▼

Yes, verl supports Megatron-LM as a training backend for models above 70B parameters with tensor and pipeline parallelism. Install the mbridge package for checkpoint format conversion and set actor_rollout_ref.model.backend=megatron in the config.

Why does verl run out of memory during rollout?▼

Rollout OOM typically comes from large log_prob_micro_batch_size or high vLLM gpu_memory_utilization. Reduce the micro batch size, enable gradient checkpointing, lower gpu_memory_utilization to 0.7, or switch to FSDP2 with CPU offloading.

Which vLLM versions are compatible with verl?▼

verl requires vLLM versions between 0.8.2 and 0.12.0; the 0.7.x series has known bugs and should be avoided. For issues on vLLM 0.8.x, set enforce_eager=true to disable CUDA graphs.

When should I not use verl for LLM post-training?▼

verl is not ideal if you need Megatron-native training (consider slime or miles), PyTorch-native Monarch abstractions (torchforge), or only simple SFT/DPO, where TRL or Axolotl are lighter-weight choices.

Related Skills

Explore More Software Engineering Skills→
CxxxxDxxxFCxxxxDxxxF

book-sft-pipeline

Convert books into SFT datasets and train LoRA models.

Community
Advanced
rabbicserabbicse

fine-tuning-expert

Optimizes LLM fine-tuning workflows with LoRA, QLora, and PEFT.

Community
Advanced
wanshuiyinwanshuiyin
15.6k

training-check

Monitors WandB training metrics periodically to detect NaN, divergence, and idle GPUs.

Community
Intermediate
NousResearchNousResearch
239.1k

axolotl

Configure and execute LLM fine-tuning workflows using Axolotl YAML configs.

Official
Advanced