skills.rest
qcmuu
qcmuuCOMMUNITY
Explore all skills by this author →
GitHub

verl-rl-training

Orchestrate verl-based RL training loops for LLM post-training.

Updated May 2, 2026
Search Tags:#reinforcement learning#distributed training#reward functions#rlhf#grpo#ppo training#vllm rollout
One-click install
npx skills add https://github.com/qcmuu/AI-Research-Skills --skill verl-rl-training-qcmuu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/qcmuu/AI-Research-Skills/tree/main/06-post-training/verl
Command: npx skills add https://github.com/qcmuu/AI-Research-Skills --skill verl-rl-training-qcmuu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0, and includes references (resource) components.

What problem does it solve?

This Skill helps you train large language models using reinforcement learning methods (RLHF/GRPO/PPO and related variants) efficiently across distributed infrastructure, without stitching together complex rollout, reward, and training components manually.

Core Features & Use Cases

  • HybridFlow RL training orchestration: coordinates rollout → reward computation → policy/value updates in a production-ready training loop.
  • Backend-flexible scaling: supports major training and rollout backends such as FSDP/FSDP2/Megatron-LM for training and vLLM/SGLang/Transformers for rollout.
  • Algorithm coverage for post-training: implements GRPO, PPO (GAE), and multiple reward/baseline strategies suitable for sparse or dense reward settings.
  • Use case: fine-tune a math reasoning model on GSM8K-style tasks by sampling multiple responses per prompt and optimizing with GRPO using a custom reward that checks extracted boxed answers.

Quick Start

Run a GRPO math training job with verl by executing a Python invocation that sets adv_estimator=grpo, points to your parquet dataset, selects a HuggingFace base model, sets rollout samples per prompt, and configures the GPU count.

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large language models with GRPO or PPO using distributed GPUs?▼

To train large language models with GRPO or PPO using distributed GPUs, you orchestrate verl-based rollout, reward computation, and policy updates by configuring the algorithm estimator, rollout backend, and GPU resources.

Can I use vLLM for rollout during PPO training?▼

Yes, you can use vLLM for rollout during PPO training. The system supports major rollout backends including vLLM, SGLang, and Transformers, coordinating rollout, reward computation, and policy updates within a production-ready training loop.

What reinforcement learning algorithms are supported for RLHF post-training?▼

Reinforcement learning algorithms supported for RLHF post-training include GRPO and PPO with GAE. These algorithms implement multiple reward and baseline strategies suitable for both sparse and dense reward settings.

Do I need Ray and PyTorch to run distributed RLHF training?▼

Yes, you need Ray and PyTorch to run distributed RLHF training. A compatible setup requires verl, torch, ray, vllm, and transformers, ensuring proper orchestration of distributed GPU resources and rollout operations.

How does reinforcement learning fine-tuning work for math reasoning tasks?▼

Reinforcement learning fine-tuning for math reasoning tasks works by sampling multiple responses per prompt and optimizing the policy with GRPO using a custom reward function that checks extracted boxed answers against ground truth.

Can I train vision-language models with reward functions using this setup?▼

Yes, you can train vision-language models with reward functions using this setup. The orchestration supports multi-turn tool-call workflows and vision-language reward training across scenarios like math reasoning and large-scale batch generation.

Related Skills

Explore More Education & Research Skills→
hhhi21ghhhi21g

verl-rl-training

Train large language models with reinforcement learning on distributed infrastructure.

Community
Advanced
ScientiaCapitalScientiaCapital
1

model-deployment

Export fine-tuned models to GGUF and deploy with vLLM or Ollama.

Community
Advanced
GregsGreyCodeGregsGreyCode
6

slime-rl-training

Coordinate end-to-end RL training for LLMs with Megatron and SGLang.

Community
Advanced
ovachieverovachiever
52

openrlhf-training

Train large language models with RLHF using Ray and vLLM acceleration.

Community
Advanced