nemo-gym-debugging

What problem does it solve?

Nemo Gym runs and reward profiling jobs can fail in ways that look similar at the top level (empty outputs, nested 500s, schema/verifier errors, stalled serving, or stale cache), but the root cause lives in a specific layer.

Core Features & Use Cases

• Layered failure classification: Guides you to distinguish infra (Slurm/Ray/container/filesystem), model serving (vLLM readiness/throughput), config, data/schema, verifier/runtime, cache/resume issues, and throughput bottlenecks.
• Deterministic debug order: Provides a step-by-step escalation ladder that prioritizes high-signal checks (job state/logs, vLLM readiness, server readiness, tool/sandbox readiness, materialized vs source timestamps, output/profiling counts, verifier exceptions).
• Targeted offline tooling for tool-call schema: Offers a bundled static checker script for Nemo Gym tool-call JSONL to catch vLLM/Outlines grammar/schema incompatibilities before expensive rollouts.
• References for copy-paste diagnostics: Includes request-boundary visibility and reusable log/JSONL inspection snippets to recover identity and evidence when nested errors hide the true provider/verifier response.

Quick Start

Run the tool-call JSONL static sanity check to quickly rule out schema/grammar incompatibilities before re-running your Nemo Gym job.