skills.rest
wshobson
wshobsonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

grpo-rlvr-training

Configures GRPO reinforcement learning with verifiable rewards for reasoning model training.

39.3k|4.2k|Updated Jul 24, 2025
Search Tags:#reward functions#trl#grpo#reasoning models#Reinforcement Learning#RLVR#llm finetuning
One-click install
npx skills add https://github.com/wshobson/agents --skill grpo-rlvr-training
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: grpo-rlvr-training
Source: https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/grpo-rlvr-training
Command: npx skills add https://github.com/wshobson/agents --skill grpo-rlvr-training

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trl, vllm, jsonschema, and includes references (resource) components.

What problem does it solve?

Training models on tasks with algorithmically checkable outcomes (math answers, code tests, schema-valid output) requires a correctly configured GRPO setup, and misconfigured reward functions or hyperparameters cause reward hacking and divergent runs.

Core Features & Use Cases

  • Reference GRPO Recipe: Provides a validated TRL GRPOTrainer configuration with vLLM-backed generation, including settled starting values for learning rate, KL coefficient, and generation count.
  • Reward Function Library: Supplies runnable reward functions for exact-match correctness, JSON schema validation, sandboxed unit-test execution, length penalties, and rubric-based judging.
  • Variant Selection Guidance: Diagnoses failure modes like entropy collapse, length bias, and MoE instability, routing to DAPO, Dr.GRPO, or GSPO variants only when symptoms appear.
  • Use Case: You have an SFT checkpoint that sometimes solves math problems correctly. Use this Skill to produce a validated GRPO config with a composite format-plus-correctness reward, inspect the reward against 50-100 sampled outputs, then hand the config to a training engineer.

Quick Start

Set up a GRPO training run with verifiable rewards for my math reasoning model that already succeeds on some problems.

Frequently Asked Questions about grpo-rlvr-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up GRPO training with TRL?▼

Use TRL's GRPOTrainer with GRPOConfig enabling vLLM generation, num_generations of at least 8, learning rate around 5e-7, and beta of 0.01. Pass a prompt-only dataset and composite reward functions covering both format and correctness.

When should I use GRPO instead of DPO or SFT?▼

Use GRPO when task success is algorithmically verifiable, such as unit tests passing or answers matching ground truth, and the model already succeeds sometimes. Use SFT when the model never succeeds, and DPO when the signal is a preference between two acceptable outputs.

What is the difference between GRPO, DAPO, Dr.GRPO, and GSPO?▼

DAPO addresses entropy collapse on long chain-of-thought by decoupling clip bounds. Dr.GRPO removes length-normalization bias when reward correlates with output length. GSPO uses sequence-level importance ratios and is required for mixture-of-experts models.

Why does my GRPO run reward-hack or diverge?▼

Reward hacking usually comes from training against an uninspected reward function that scores the wrong thing. Manually read the reward function's judgments on 50-100 sampled outputs before training, and fix disagreements with human judgment before tuning hyperparameters.

How much GPU memory does GRPO training need?▼

Small models up to roughly 3B fit on 24GB GPUs with vLLM sleep mode, 8-bit AdamW, and gradient checkpointing combined. Around 32B parameters needs an H200-class GPU, and 70B-class models need B200-class hardware due to policy, reference, and generation engine residency.

Can I run GRPO on vision-language models?▼

Vision-language RL is documented as reference-only and not executed by this plugin. Tooling is fragmented across ms-swift and EasyR1 forks, and naive text-only GRPO on a VLM tends to reward-hack by ignoring the image input.

Related Skills

Explore More Software Engineering Skills→
KappTech88KappTech88

model-merging

Merge base and finetuned models into a deployable artifact using linear, SLERP, TIES, DARE, or task arithmetic.

Community
Advanced
lucasfthlucasfth

grpo-rl-training

Implement GRPO training workflows for TRL-based reinforcement learning with language models.

Community
Advanced
marchattonmarchatton
5

validate-evaluator

Calibrate LLM judges against human labels using TPR/TNR measurements and bias correction.

Community
Advanced
Aum08DesaiAum08Desai

simpo-training

Train LLMs with reference-free SimPO preference optimization.

Community
Advanced