skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

openrlhf-training

Run distributed RLHF training with Ray orchestration and vLLM acceleration.

2|Updated Apr 12, 2026
Search Tags:#ray#distributed-training#ppo#vllm#dpo#rlhf#deepspeed
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill openrlhf-training-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openrlhf-training
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/06-post-training/openrlhf
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill openrlhf-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a scalable, high-performance RLHF training framework that reduces GPU idle time, prevents OOMs, and enables stable reinforcement learning fine-tuning of large language models across multi-node clusters.

Core Features & Use Cases

  • Distributed RLHF Workflows: End-to-end support for PPO, GRPO, RLOO, REINFORCE++, and DPO pipelines including reward model training and PPO fine-tuning.
  • vLLM + Ray Hybrid Engine: vLLM inference acceleration combined with Ray placement groups for multi-node orchestration and dynamic sleep/wake GPU sharing.
  • Memory & Checkpointing: DeepSpeed ZeRO-3 sharding, DeepSpeed sleep/offload cycles, robust checkpointing and resume for long-running experiments.
  • Use Case: Run a multi-node training job to pretrain a reward model and then perform PPO with vLLM-accelerated rollouts on models from 7B to 70B+.

Quick Start

Start an OpenRLHF PPO training run on an 8-GPU Ray cluster using vLLM acceleration and ZeRO-3 sharding.

Frequently Asked Questions about openrlhf-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run distributed RLHF training for large language models without running into OOM errors?▼

Distributed RLHF training prevents OOM errors by using DeepSpeed ZeRO-3 parameter sharding and checkpoint offloading. This approach shards model states across multi-node GPU clusters, enabling stable reinforcement learning fine-tuning for models from 7B to 70B+ parameters.

Can I use vLLM to accelerate PPO rollouts during multi-node training?▼

Yes, vLLM accelerates PPO rollouts by combining inference engine tensor-parallelism with Ray placement groups. This hybrid engine uses dynamic sleep/wake GPU sharing to reduce idle time and optimize resource utilization across multi-node clusters.

What RLHF algorithms are supported for large model fine-tuning?▼

Supported RLHF algorithms include PPO, GRPO, RLOO, REINFORCE++, and DPO workflows. The framework provides end-to-end pipeline support, covering both reward model training and subsequent reinforcement learning fine-tuning stages.

Does Ray orchestration work with DeepSpeed checkpointing for long-running experiments?▼

Ray orchestration integrates with DeepSpeed checkpointing to provide robust save and resume capabilities for long-running experiments. It manages multi-node GPU clusters while DeepSpeed handles sleep and offload cycles to maintain training state efficiently.

What's the best way to prevent GPU idle time during multi-node reinforcement learning training?▼

Preventing GPU idle time requires a hybrid vLLM and Ray engine that implements dynamic sleep and wake GPU sharing. This method ensures continuous resource utilization across multi-node clusters during both inference and training phases.

Do I need ZeRO-3 parameter sharding to train a 70B model with PPO?▼

Yes, ZeRO-3 parameter sharding is required to train 70B models with PPO across multi-node clusters. Sharding partitions optimizer states and gradients, preventing memory exhaustion and enabling stable reinforcement learning workflows.

Related Skills

Explore More Education & Research Skills→
ovachieverovachiever
52

openrlhf-training

Train large language models with RLHF using Ray and vLLM acceleration.

Community
Advanced
ScientiaCapitalScientiaCapital
1

model-deployment

Export fine-tuned models to GGUF and deploy with vLLM or Ollama.

Community
Advanced
Supporter09Supporter09

openrlhf-training

Train large language models with distributed RLHF workflows using Ray and vLLM.

Community
Advanced
Ced3-hanCed3-han

openrlhf-training

Run distributed RLHF training for large language models with Ray and vLLM.

Community
Advanced