skills.rest
agentscope-ai
agentscope-aiOFFICIAL
Explore all skills by this author →
𝕏@agentscope_aiGitHub

rl-reward

Build RL reward signals for GRPO, DPO, and Best-of-N using OpenJudge graders.

809|65|Updated Jul 8, 2025
Search Tags:#DPO#LLM-as-judge#rlhf#grpo#preference pairs#reward model#openjudge
One-click install
npx skills add https://github.com/agentscope-ai/OpenJudge --skill rl-reward
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rl-reward
Source: https://github.com/agentscope-ai/OpenJudge/tree/main/skills/rl-reward
Command: npx skills add https://github.com/agentscope-ai/OpenJudge --skill rl-reward

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires py-openjudge.

What problem does it solve?

Training language models with RLHF or RLAIF requires reliable scalar rewards or preference pairs, but building reward models from scratch is costly and LLM-as-judge scoring is noisy and inconsistent across scales. This Skill provides decision trees and code patterns for constructing pointwise, pairwise tournament, and preference-pair rewards with the OpenJudge library.

Core Features & Use Cases

  • Strategy Selection: Decision trees map your RL algorithm (GRPO, REINFORCE, DPO, IPO) and task type (verifiable vs subjective) to the right reward strategy—pointwise, pairwise tournament, or listwise ranking.
  • Pointwise Multi-Dimension Rewards: Combine graders like CorrectnessGrader, HarmfulnessGrader, and CodeExecutionGrader with WeightedSumAggregator, plus score normalization for training stability.
  • Pairwise Tournaments & Preference Pairs: Use GRPOTournamentEvaluationStrategy for net-win-rate rewards on subjective tasks, and LISTWISE-mode LLMGrader to generate (chosen, rejected) pairs for DPO with position-bias debiasing.
  • Use Case: While training a chat model with GRPO on instruction-following tasks, run all-pairs comparison across each group of rollouts to compute net win rates as relative rewards, avoiding unreliable absolute scoring.

Quick Start

Ask the AI to build a GRPO reward function for subjective tasks using OpenJudge's pairwise tournament strategy with debiasing enabled.

Frequently Asked Questions about rl-reward

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a reward function for GRPO with an LLM judge?▼

For verifiable tasks like math or code, use pointwise graders such as StringMatchGrader or CodeExecutionGrader for exact scores. For subjective tasks, use GRPOTournamentEvaluationStrategy to compare all rollout pairs and compute net win rate as the reward.

How to generate DPO preference pairs with OpenJudge?▼

Create an LLMGrader in GraderMode.LISTWISE with a comparison template, then evaluate pairs of responses. A rank of [1, 2] means response_1 is chosen; use GradingRunner to parallelize labeling across a large dataset.

Pointwise vs pairwise reward: which should I use for RLHF?▼

Use pointwise for verifiable tasks where absolute scoring works, and pairwise tournament for subjective tasks where relative comparison is more reliable. Pairwise costs N*(N-1)/2 judge calls per group of N rollouts.

How do I reduce LLM judge noise in reward scoring?▼

Wrap graders with VotingEvaluationStrategy for discrete scores or AverageEvaluationStrategy for continuous scores to aggregate multiple calls. For pairwise comparisons, enable debiased mode to run both orderings and discard inconsistent results.

Why do my RL rewards need normalization before training?▼

OpenJudge LLM graders return scores on a 1–5 scale while function graders return 0–1, so mixing them creates imbalanced reward signals. Normalize all scores to a common 0–1 range before aggregation to keep training stable.

What is the cost of pairwise tournament rewards for large GRPO groups?▼

A group of N rollouts requires N*(N-1)/2 comparisons, doubling with debiasing—for example, 8 rollouts need 28 calls or 56 debiased. Keep group size at 8 or fewer and use a small judge model for cost-sensitive training.

Related Skills

Explore More Software Engineering Skills→
GratefulJinx77GratefulJinx77

evals

Evaluate AI agents using code, model, and human graders.

Community
Advanced
kaminocorpkaminocorp
228

grpo-rl-training

Implement GRPO training with TRL for language-model fine-tuning.

Official
Advanced
Thiago-Cruz-engThiago-Cruz-eng

agent-evaluation

Design evaluation systems for AI agents with benchmarks and grader rubrics.

Community
Advanced
thistleknotthistleknot

synthetic-data

Generate synthetic training datasets for LLM fine-tuning with distilabel pipelines.

Community
Advanced