skills.rest
Jensen-Yao
Jensen-YaoCOMMUNITY
Explore all skills by this author →
GitHub

openrlhf-training

Trains large language models with distributed PPO, GRPO, RLOO, and DPO using Ray and vLLM.

2|Updated Aug 15, 2026
Search Tags:#ray#vLLM#RLHF#distributed training#PPO#GRPO#openrlhf
One-click install
npx skills add https://github.com/Jensen-Yao/agents-skills --skill openrlhf-training-jensen-yao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openrlhf-training
Source: https://github.com/Jensen-Yao/agents-skills/tree/main/skills/openrlhf
Command: npx skills add https://github.com/Jensen-Yao/agents-skills --skill openrlhf-training-jensen-yao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openrlhf, ray, vllm, torch, transformers, deepspeed, and includes references (resource) components.

What problem does it solve? RLHF post-training of large models (7B-70B+) requires coordinating actor, critic, reward, and reference models across GPU clusters while keeping generation fast and memory under control. This Skill provides ready-to-use OpenRLHF commands and workflows for distributed RLHF training with Ray scheduling and vLLM inference acceleration. ## Core Features & Use Cases - Distributed PPO/GRPO/RLOO Training: Launch Ray-based training jobs with colocated models, vLLM generation engines, and ZeRO-3 sharding for models up to 70B+ parameters. - Reward Model and DPO Pipelines: Train reward models with DeepSpeed or run DPO directly on preference datasets without a reward model. - Hybrid Engine GPU Sharing: Use vLLM and DeepSpeed sleep modes to share GPUs across all models, cutting GPU requirements by roughly 25%. - Use Case: You want to run GRPO on a Llama-3-8B SFT checkpoint across 8 A100 GPUs. The Skill gives you the exact ray job submit command with group_norm advantage estimation, KL loss settings, and memory-efficient colocation flags. ## Quick Start Ask the agent to set up an OpenRLHF GRPO training job for your SFT model on your available GPUs, including the Ray cluster launch and training command.

Frequently Asked Questions about openrlhf-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with GRPO using OpenRLHF?▼

Run the train_ppo_ray module with --advantage_estimator group_norm to enable GRPO, which removes the need for a critic model. Add --use_kl_loss with --kl_estimator k3 for the KL loss from the GRPO paper, and optionally --no_advantage_std_norm to disable standard deviation normalization.

What is the difference between PPO, GRPO, and RLOO in OpenRLHF?▼

PPO uses a critic network for maximum stability but consumes more memory, while GRPO and RLOO are critic-free and about 30% lighter. RLOO adds per-token KL rewards with PPO-clip, and GRPO normalizes rewards within groups; REINFORCE++ is the most stable critic-free option.

OpenRLHF vs TRL vs veRL for RLHF training?▼

OpenRLHF targets distributed multi-node training of 7B-70B+ models with Ray and vLLM acceleration, running about 2x faster than DeepSpeedChat. TRL suits simpler single-node training, while veRL is ByteDance's framework aimed at very large models like 671B.

How many GPUs do I need to train a 70B model with OpenRLHF?▼

A 70B model with PPO typically needs 48 A100 80GB GPUs using the Hybrid Engine with a 1:1:1 vLLM-to-actor-to-critic ratio. Without colocation the same setup requires 64 GPUs, so enabling --colocate_all_models with sleep modes saves roughly 25% of hardware.

Why does OpenRLHF give a DeepSpeed GPU index out of range error?▼

This error occurs because Ray modifies CUDA_VISIBLE_DEVICES in a way DeepSpeed does not expect. Fix it by setting the environment variable RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 before launching the training job.

Can I use a custom reward function instead of a reward model in OpenRLHF?▼

Yes, pass a Python file via --remote_rm_url implementing a reward_func that receives queries, prompts, and labels and returns rewards, scores, and extra logs. This supports reinforced fine-tuning for tasks like code execution testing or math answer verification.

Related Skills

Explore More Software Engineering Skills→
zerwizzerwiz

axolotl

Configure and troubleshoot Axolotl YAML training workflows for LLM fine-tuning.

Community
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
Supporter09Supporter09

openrlhf-training

Train large language models with distributed RLHF workflows using Ray and vLLM.

Community
Advanced
davila7davila7
30.5k

verl-rl-training

Train LLMs with reinforcement learning using verl for RLHF, GRPO, and PPO post-training.

Community
Advanced