openrlhf-training

Orchestrate distributed RLHF training for large language models with Ray and vLLM.

Updated Aug 23, 2026

Search Tags:#ray #distributed-training #ppo #vllm #dpo #rlhf #grpo

Ready to run

npx skills add https://github.com/t2ance/dr-claw-plugin --skill openrlhf-training-t2ance

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: openrlhf-training
Source: https://github.com/t2ance/dr-claw-plugin/tree/main/plugins/ml-training-stack/skills/post-training/openrlhf
Command: npx skills add https://github.com/t2ance/dr-claw-plugin --skill openrlhf-training-t2ance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openrlhf, ray, vllm, torch, transformers, deepspeed, and includes references (resource) components.

What problem does it solve?

OpenRLHF enables scalable, distributed RLHF training for large language models using Ray, vLLM, and ZeRO-3, accelerating PPO/GRPO/DPO/RLOO workflows.

Core Features & Use Cases

Distributed RLHF training across multiple GPUs and nodes with hybrid resource sharing.
Supports PPO, GRPO, RLOO, DPO with Ray-based orchestration and vLLM inference acceleration.
Checkpointing, multi-node scheduling, and GPU memory management for large models.

Quick Start

Run a quick PPO training example on a multi-node cluster to verify setup.

Frequently Asked Questions about openrlhf-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run distributed RLHF training across multiple nodes?▼

Distributed RLHF training across multiple nodes is orchestrated using Ray for scheduling and vLLM for inference acceleration, leveraging ZeRO-3 and hybrid engine sharing to manage GPU resources.

Does this support PPO, GRPO, and DPO workflows for large language models?▼

Yes, it supports PPO, GRPO, RLOO, and DPO workflows for large language models, utilizing Ray-based orchestration and vLLM inference acceleration to scale training efficiently.

What dependencies do I need for multi-node PPO and GRPO training?▼

You need openrlhf, ray, vllm, torch, transformers, and deepspeed installed to enable multi-node, GPU-accelerated PPO and GRPO training with ZeRO-3 optimization.

How does vLLM accelerate inference during RLHF training?▼

vLLM accelerates inference during RLHF training by acting as the high-performance generation engine, paired with Ray orchestration to handle large model workloads and optimize GPU memory management.

Can I use ZeRO-3 for GPU memory management in distributed training?▼

Yes, ZeRO-3 is supported for GPU memory management during distributed training, allowing you to handle large models through memory optimization alongside checkpointing and sleep modes.

What is the best way to checkpoint large models during distributed RLHF?▼

The best way to checkpoint large models during distributed RLHF is using the built-in checkpointing functionality, which preserves training states across multi-node GPU clusters orchestrated by Ray.