skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

verl-rl-training

Train large language models with reinforcement learning using verl and Ray.

2|Updated Apr 12, 2026
Search Tags:#distributed-training#reinforcement-learning#ppo#vllm#rlhf#grpo#verl
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill verl-rl-training-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/06-post-training/verl
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill verl-rl-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provides practical, production-ready guidance to perform post-training reinforcement learning (RL) on large language models, removing guesswork around distributed rollouts, reward design, and backend interoperability so teams can reliably optimize model behavior at scale.

Core Features & Use Cases

  • Production-grade workflows for RLHF, GRPO, PPO and related algorithms targeting models from small to 671B parameters.
  • Flexible backend and rollout integrations (vLLM, SGLang, FSDP, Megatron) for single-node and multi-node training.
  • Examples and troubleshooting for math reasoning, vision-language RL, LoRA RL, critic-based PPO, and Megatron expert parallelism.

Quick Start

Launch a GRPO training run with verl using your parquet dataset, a vLLM rollout, and an 8-sample rollout configuration.

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scale RLHF training for large language models across multiple GPU nodes?▼

To scale RLHF training across multiple GPU nodes, you can use Ray orchestration with verl to coordinate distributed multi-node training. This setup supports models up to 671B parameters and integrates with FSDP or Megatron backends for production deployments.

Can I use vLLM for rollout generation during PPO training?▼

Yes, you can use vLLM as a rollout engine during PPO training. The system supports flexible backend and rollout integrations including vLLM and SGLang to handle reward models and rollout engines efficiently during reinforcement learning.

What's the best way to start a GRPO training run with a parquet dataset?▼

The best way to start GRPO training is to launch a run using verl with your parquet dataset, configuring a vLLM rollout and an 8-sample rollout setup. This provides a quick start for post-training optimization.

Does verl support reinforcement learning for vision-language agents and math reasoning tasks?▼

Yes, verl supports reinforcement learning for vision-language agents and math reasoning tasks. It provides specific examples and troubleshooting for these scenarios, alongside LoRA RL and critic-based PPO implementations.

Do I need Ray orchestration to handle KL-control settings in distributed LLM training?▼

Yes, Ray orchestration is required to manage distributed multi-node training and handle KL-control settings. This orchestration ensures reliable backend interoperability with PyTorch and optional Megatron during large-scale reinforcement learning.

Why does distributed RL training fail when switching between FSDP and Megatron backends?▼

Distributed RL training can fail during backend switches due to interoperability issues between FSDP and Megatron. The system provides troubleshooting for backend integrations and Megatron expert parallelism to resolve these deployment conflicts.

Related Skills

Explore More Software Engineering Skills→
AscendAscend
31

verl-deploy

Deploys Verl training services on NPU clusters with RocketMQ, SeaWeedFS, and OpenResty.

Official
Advanced
vasilyu1983vasilyu1983
73

ai-llm-inference

Optimize LLM inference performance and cost with vLLM and quantization.

Community
Advanced
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
jonnabiojonnabio
6

verl-rl-training

Train LLMs with reinforcement learning algorithms using verl.

Community
Advanced