skills.rest
MedTiLab
MedTiLabCOMMUNITY
Explore all skills by this author →
GitHub

openrlhf-training

Train large language models with distributed PPO, GRPO, RLOO, and DPO using Ray and vLLM.

5|2|Updated Aug 10, 2026
Search Tags:#ray#vLLM#RLHF#distributed training#PPO#GRPO#openrlhf
One-click install
npx skills add https://github.com/MedTiLab/Auto-meta-analysis --skill openrlhf-training-medtilab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openrlhf-training
Source: https://github.com/MedTiLab/Auto-meta-analysis/tree/main/skills/post-training/openrlhf
Command: npx skills add https://github.com/MedTiLab/Auto-meta-analysis --skill openrlhf-training-medtilab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openrlhf, ray, vllm, torch, transformers, deepspeed, and includes references (resource) components.

What problem does it solve? RLHF training of large models (7B-70B+) requires coordinating Actor, Critic, Reward, and Reference models across GPU clusters while keeping generation fast and memory usage under control. This Skill provides ready-to-use OpenRLHF commands and workflows for distributed RLHF with Ray scheduling and vLLM inference acceleration. ## Core Features & Use Cases - Distributed PPO/GRPO/RLOO Training: Launch Ray-based RLHF jobs with Hybrid Engine GPU sharing, vLLM sleep mode, and DeepSpeed ZeRO-3 sharding. - Reward Model and DPO Training: Train reward models or run direct preference optimization without a reward model using DeepSpeed. - Custom Rewards and Agent RLHF: Implement custom reward functions for code generation or math reasoning, and build multi-step agent training loops. - Use Case: Fine-tune a Llama-3-8B model on preference data by training a reward model, then running PPO with colocated models on 8 A100 GPUs, cutting GPU idle time through sleep/wake cycles. ## Quick Start Use the openrlhf-training skill to set up a GRPO training job for a Llama-3-8B model on my 8-GPU Ray cluster.

Frequently Asked Questions about openrlhf-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with GRPO using OpenRLHF?▼

Run train_ppo_ray with --advantage_estimator group_norm to enable GRPO, which removes the need for a critic model. Add --use_kl_loss with --kl_estimator k3 for the KL loss from the GRPO paper, and --no_advantage_std_norm to disable standard deviation normalization.

What is the difference between PPO, GRPO, and RLOO in OpenRLHF?▼

PPO uses a critic network for maximum stability but higher memory. GRPO normalizes rewards within groups without a critic, while RLOO uses leave-one-out baselines with per-token KL rewards. REINFORCE++ offers PPO-level stability without a critic and trains about 25% faster.

How many GPUs do I need to train a 70B model with RLHF?▼

A 70B model requires roughly 48 A100 80GB GPUs with the Hybrid Engine enabled, using a vLLM:Actor:Critic ratio of 1:1:1. Without model colocation, you need about 64 dedicated GPUs since each component gets its own allocation.

Why does OpenRLHF fail with a DeepSpeed GPU index out of range error?▼

This error occurs when Ray sets CUDA_VISIBLE_DEVICES in a way DeepSpeed cannot map. Fix it by setting the environment variable RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 before launching the training job.

How do I fix GPU OOM during OpenRLHF PPO training?▼

Remove --colocate_all_models so each model gets dedicated GPUs, or lower --vllm_gpu_memory_utilization from 0.5 to 0.4. You can also reduce micro batch sizes, enable --vllm_enable_sleep and --deepspeed_enable_sleep, and use --zero_stage 3 for maximum parameter sharding.

When should I use TRL or veRL instead of OpenRLHF?▼

Use TRL for single-node training with a simpler API, and veRL for extremely large models around 671B parameters. OpenRLHF fits best when you need distributed multi-node training with vLLM acceleration and multiple RL algorithms in one framework.

Related Skills

Explore More Software Engineering Skills→
OpenLAIROpenLAIR
1.0k

fine-tuning-with-trl

Automates end-to-end LLM fine-tuning workflows with TRL and RLHF techniques.

Official
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
plurigridplurigrid
60

fokker-planck-analyzer

Compare empirical weight distributions to Gibbs distributions for Langevin training convergence.

Official
Advanced
azaanalirazaazaanaliraza

fine-tuning-with-trl

Fine-tune language models with TRL using SFT, DPO, PPO, and GRPO training methods.

Community
Advanced