verl-rl-training

What problem does it solve?

This Skill addresses the complexity and resource demands of training Large Language Models (LLMs) using Reinforcement Learning (RL) at scale, providing a robust framework for advanced post-training techniques.

Core Features & Use Cases

• Scalable RL Training: Supports training up to 671B parameter models using distributed backends like FSDP and Megatron-LM.
• Flexible RL Algorithms: Implements PPO, GRPO, RLOO, REINFORCE++, and more, with support for custom reward functions.
• Backend Agnosticism: Seamlessly switch between rollout engines (vLLM, SGLang) and training backends (FSDP, Megatron-LM).
• Use Case: Fine-tune a Llama-3 model using GRPO on a math reasoning dataset to improve its problem-solving capabilities, leveraging a multi-GPU cluster for efficient training.

Quick Start

Use the verl-rl-training skill to launch a GRPO training job for a Qwen2.5-7B model on the GSM8K dataset using 8 GPUs.