What problem does it solve?
This Skill eliminates the guesswork and repetitive setup required to add a new benchmark or training/evaluation environment to NeMo-Gym, especially when you must wire data, a resources server, an agent, and verification logic into one working workflow.
Core Features & Use Cases
- Benchmark integration workflow: Covers adding either a native benchmark (resources server implements
verify() and reward logic) or an external benchmark (wraps an existing library at the agent layer) end to end.
- Data + verifier schema guidance: Shows how to convert datasets into Gym JSONL with
responses_create_params.input and verifier_metadata, including generating example.jsonl, validating data, and preparing train/validation via GitLab registry.
- Robust
verify() implementation rules: Provides guardrails for safe reward computation (binary rewards), concurrency control, graceful failure on empty/bad outputs, and patterns for subprocess/ray execution and external tool auto-install.
- Config and testing checklist: Guides YAML wiring of resources server + agent pairing(s), required dataset fields, and how to run server tests and do smoke/e2e rollouts, followed by reward baselining/profiling.
Quick Start
Ask the assistant to guide you through adding a new benchmark by having it walk you through scaffolding a NeMo-Gym resources server, defining your JSONL data schema and verify() reward logic, wiring the YAML config, and running tests plus a baseline reward profile.
