skills.rest
MedTiLab
MedTiLabCOMMUNITY
Explore all skills by this author →
GitHub

torchforge-rl-training

Implements PyTorch-native agentic RL training with GRPO, Monarch, and TorchTitan.

5|2|Updated Aug 10, 2026
Search Tags:#PyTorch#reinforcement learning#LLM training#vLLM#distributed training#GRPO#torchtitan
One-click install
npx skills add https://github.com/MedTiLab/Auto-meta-analysis --skill torchforge-rl-training-medtilab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: torchforge-rl-training
Source: https://github.com/MedTiLab/Auto-meta-analysis/tree/main/skills/post-training/torchforge
Command: npx skills add https://github.com/MedTiLab/Auto-meta-analysis --skill torchforge-rl-training-medtilab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.9.0, torchtitan>=0.2.0, vllm, monarch, and includes references (resource) components.

What problem does it solve? Building reinforcement learning training pipelines for LLMs typically requires wiring together distributed training, inference, and weight synchronization infrastructure by hand. This Skill provides guidance for using torchforge, Meta's PyTorch-native RL library, so you can focus on algorithm design while the framework handles distributed coordination. ## Core Features & Use Cases - GRPO and SFT Training Workflows: Step-by-step configuration and launch instructions for math reasoning training with group-relative advantages and supervised fine-tuning. - Custom Loss Functions: Implement new RL algorithms (GRPO, DAPO, SAPO variants) in roughly 100 lines using built-in loss classes like SimpleGRPOLoss and ReinforceLoss. - Distributed Scaling: Configure multi-GPU and multi-node training with Monarch actors, TorchTitan FSDP parallelism, and vLLM inference. - Use Case: Train a Qwen2.5-7B model on GSM8K with GRPO across 3 GPUs by defining a YAML config, a math reward function, and launching with a single command. ## Quick Start Ask the AI to help you set up a GRPO training run with torchforge for a math reasoning model, including the YAML configuration and reward function.

Frequently Asked Questions about torchforge-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with GRPO using torchforge?▼

Create a YAML config specifying the model, dataset, and GRPO parameters like n_samples and beta, define a reward function, then launch with python -m apps.grpo.main --config your_config.yaml. GRPO training requires at least 3 GPUs for the trainer, generator, and reference model services.

What is torchforge and when should I use it for RL training?▼

torchforge is Meta's PyTorch-native RL library that separates infrastructure from algorithms, letting you implement methods like GRPO or DAPO in about 100 lines. Use it for algorithm experimentation; for production stability, consider miles or verl instead.

How many GPUs does torchforge GRPO training require?▼

SFT requires at least 2 GPUs (trainer and generator), while GRPO requires at least 3 GPUs to run the trainer, vLLM generator, and frozen reference model. You can reduce requirements by running the reference model on CPU or colocating services.

Why does torchforge training run out of memory during generation?▼

CUDA OOM in vLLM typically comes from large batch sizes or long sequences. Reduce n_samples in the GRPO config, lower seq_len, or decrease vLLM's gpu_memory_utilization to around 0.7.

How do I fix policy collapse during GRPO training?▼

Policy collapse shows as entropy dropping to zero with stalled rewards. Increase the KL penalty coefficient beta, add an entropy bonus via entropy_coef, or reduce the learning rate to stabilize training.

Is torchforge stable enough for production RL pipelines?▼

No, torchforge and Monarch are experimental and APIs may change between versions. The documentation recommends miles or verl for production-ready stability, or slime if you need Megatron-native training.

Related Skills

Explore More Software Engineering Skills→
MLT-OSSMLT-OSS
14

evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Official
Advanced
ricablericable

gptq

Compress large language models into GPTQ 4-bit checkpoints.

Community
Advanced
plurigridplurigrid
60

fokker-planck-analyzer

Compare empirical weight distributions to Gibbs distributions for Langevin training convergence.

Official
Advanced
MesferAliMesferAli

slime-rl-training

Train GLM models with RL using Megatron-LM and SGLang.

Community
Advanced