skills.rest
JamesFincher
JamesFincherCOMMUNITY
Explore all skills by this author →
GitHub

slime-rl-training

Combine Megatron-LM training with SGLang rollouts for LLM reinforcement learning.

Updated May 4, 2026
Search Tags:#reinforcement learning#distributed training#sglang#grpo#megatron-lm#llm training#rl post-training
One-click install
npx skills add https://github.com/JamesFincher/gengar --skill slime-rl-training-jamesfincher
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: slime-rl-training
Source: https://github.com/JamesFincher/gengar/tree/main/optional-skills/mlops/slime
Command: npx skills add https://github.com/JamesFincher/gengar --skill slime-rl-training-jamesfincher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sglang-router>=0.2.3, ray, torch>=2.0.0, transformers>=4.40.0, and includes references (resource) components.

What problem does it solve?

slime-rl-training streamlines the difficult post-training workflow of reinforcement learning for large language models by pairing Megatron-LM training with high-throughput rollouts generated through SGLang.

Core Features & Use Cases

  • Megatron-LM RL Training: Runs actor/critic-style optimization with full parallelism (TP/PP/DP/SP) to scale training throughput.
  • SGLang Rollouts via Router: Generates rollout samples efficiently using SGLang and a router, enabling scalable reward collection.
  • Data Buffer & On/Off-Policy Workflows: Manages prompts, generated samples, filtering, and optional off-policy buffering to support practical experimentation and throughput tuning.
  • Use Case: Post-train a GLM or similar model with GRPO-style advantage estimation using JSONL prompt/label data while routing rollout generation for steady GPU utilization.

Quick Start

Run GRPO training using the Qwen3 model configuration by executing the instruction: Source the model config with the command "source scripts/models/qwen3-4B.sh" and then start training with "python train.py --actor-num-nodes 1 --actor-num-gpus-per-node 4 --rollout-num-gpus 4 --advantage-estimator grpo --use-kl-loss --kl-loss-coef 0.001 --rollout-batch-size 32 --n-samples-per-prompt 8 --global-batch-size 256 --num-rollout 3000 --prompt-data /path/to/data.jsonl ${MODEL_ARGS[@]} ${CKPT_ARGS[@]}".

Frequently Asked Questions about slime-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run reinforcement learning post-training for LLMs using Megatron-LM and SGLang?▼

Reinforcement learning post-training pairs Megatron-LM training with SGLang-based rollout generation. You source a model config and run train.py with arguments specifying actor GPUs, rollout GPUs, advantage estimator, batch sizing, and prompt data.

Can I use GRPO for advantage estimation during LLM RL training?▼

Yes, GRPO is supported as an advantage estimator. You enable it by passing the --advantage-estimator grpo CLI argument, alongside optional KL loss controls and batch sizing configurations.

Does slime-rl-training support multi-turn agentic rollouts and custom reward pipelines?▼

Yes, the Skill applies multi-turn agentic rollouts and supports custom reward or data-generation pipelines. It manages prompts, generated samples, and filtering through a data buffer.

What models are compatible with this Megatron-LM and SGLang RL training setup?▼

The setup applies to GLM, Qwen, DeepSeek, and Llama-family models. You must configure the Megatron training environment and align CLI arguments for the specific model.

Do I need Ray and PyTorch to perform distributed RL training with SGLang rollouts?▼

Yes, Ray and PyTorch are required dependencies. You also need Transformers, an SGLang router, and a configured Megatron-LM training environment to scale training throughput.

Related Skills

Explore More Software Engineering Skills→
Z43LZ43L

slime-rl-training

Integrates Megatron-LM training loops with accelerated SLang rollouts for LLM RL fine-tuning.

Community
Advanced
SethyPagnaSethyPagna

slime-rl-training

Automate reinforcement-learning post-training setup for large language models.

Community
Advanced
qcmuuqcmuu

slime-rl-training

Coordinate Megatron-LM training with SGLang rollouts for LLM reinforcement-learning post-training.

Community
Advanced
zangjeicyzangjeicy

slime-rl-training

Train LLMs with GRPO and PPO using Megatron-LM and SGLang.

Community
Advanced