skills.rest
hhhi21g
hhhi21gCOMMUNITY
Explore all skills by this author →
GitHub

openrlhf-training

Train large language models with RLHF using Ray and vLLM.

Updated Apr 11, 2026
Search Tags:#training#RLHF#large language models#text generation#language understanding
One-click install
npx skills add https://github.com/hhhi21g/HealthCenter --skill openrlhf-training-hhhi21g
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openrlhf-training
Source: https://github.com/hhhi21g/HealthCenter/tree/main/.codex/skills/openrlhf
Command: npx skills add https://github.com/hhhi21g/HealthCenter --skill openrlhf-training-hhhi21g

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openrlhf, ray, vllm, torch, transformers, deepspeed, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill enables efficient training and optimization of large language models using reinforcement learning from human feedback (RLHF), allowing for high-performance and resource-efficient training of models.

Core Features & Use Cases

  • High-Performance RLHF Training: Utilizes Ray and vLLM for distributed training with accelerated inference.
  • Large Model Training: Supports training models with sizes up to 70B parameters.
  • Algorithm Support: Offers a variety of reinforcement learning algorithms including PPO, GRPO, RLOO, and DPO.
  • Use Case: Ideal for researchers and engineers looking to train large language models for tasks such as text generation, language understanding, and question answering.

Quick Start

Train a large language model using the openrlhf-training skill with the following command:

ray start --head --node-ip-address 0.0.0.0 --num-gpus 8
ray job submit --address="http://127.0.0.1:8265" \
  -- python3 -m openrlhf.cli.train_ppo_ray \
  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \
  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
  --colocate_all_models \
  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
  --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \
  --save_path ./output/llama3-8b-rlhf \
  --micro_train_batch_size 8 --train_batch_size 128 \
  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \
  --zero_stage 3 --bf16 \
  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \
  --init_kl_coef 0.01 --normalize_reward \
  --vllm_enable_sleep --deepspeed_enable_sleep

Frequently Asked Questions about openrlhf-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large language models with RLHF using Ray and vLLM?▼

Train large language models with RLHF by utilizing Ray for distributed training and vLLM for accelerated inference across multiple GPUs. This approach supports models up to 70B parameters for text generation tasks.

What RLHF algorithms are supported for large model training?▼

Supported RLHF algorithms for large model training include PPO, GRPO, RLOO, and DPO. These algorithms optimize language models for tasks like text generation and question answering.

Can I use DeepSpeed and vLLM for distributed RLHF training?▼

Yes, you can use DeepSpeed and vLLM for distributed RLHF training. DeepSpeed manages memory with zero stage 3 optimization, while vLLM accelerates inference during the model rollout process.

Do I need multiple GPUs to run RLHF training for large language models?▼

Yes, RLHF training for large language models requires multiple GPUs. The distributed setup uses Ray to allocate resources across reference, reward, critic, and actor models with specified GPU counts.

What is the best way to optimize memory during 70B parameter RLHF training?▼

Optimize memory during 70B parameter RLHF training by enabling bf16 precision, DeepSpeed zero stage 3, and vLLM sleep mode. These features reduce memory overhead during distributed training.

How does vLLM acceleration work in reinforcement learning from human feedback?▼

vLLM acceleration in reinforcement learning from human feedback speeds up the generation of responses during the rollout phase. It uses tensor parallelism to efficiently process batches across multiple GPUs.

Related Skills

Explore More Software Engineering Skills→
OpenLAIROpenLAIR
1.0k

openrlhf-training

Run distributed RLHF training for large language models on Ray clusters.

Official
Advanced
gagan114662gagan114662

blip-2-vision-language

Generate image captions and answer visual questions with BLIP-2.

Community
Intermediate
davpatel605-beepdavpatel605-beep

huggingface-accelerate

Configure distributed PyTorch training across GPUs and nodes.

Community
Advanced
jonnabiojonnabio
6

openrlhf-training

Train large language models with RLHF algorithms using distributed GPU resources.

Community
Advanced