skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

launching-evals

Run, monitor, and debug LLM evaluation jobs on Slurm with nemo-evaluator-launcher.

Updated May 23, 2026
Search Tags:#debugging#llm evaluation#log analysis#performance monitoring#slurm#nemo-evaluator-launcher#artifact inspection
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill launching-evals-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: launching-evals
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Model-Optimizer/launching-evals
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill launching-evals-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It helps you run LLM evaluation jobs reliably on Slurm, monitor them in real time, and quickly diagnose failures by correlating invocation status with the right client/server logs and artifacts.

Core Features & Use Cases

  • Launch and control evaluations: start eval runs with nemo-evaluator-launcher, limit samples, and preview resolved configs.
  • Monitor progress with correctness: check invocation/task status and avoid falsely assuming server health from RUNNING state.
  • Debug failed runs end-to-end: pivot to both client logs and server logs to find the true root cause, then verify via dry-run/smoke tests.
  • Extract and analyze results: locate output directories and artifacts, then follow benchmark-specific analysis guidance to interpret metrics.

Quick Start

Use the launching-evals skill to run an evaluation, then monitor its invocation until it finishes, starting from a config you already have.

Frequently Asked Questions about launching-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug failed LLM evaluation jobs on Slurm?▼

To debug failed LLM evaluation jobs on Slurm, pivot to both client and server logs to find the true root cause, then verify fixes via dry-run or smoke tests before resuming the invocation.

Why does my nemo-evaluator-launcher job show RUNNING but the server is not ready?▼

A Slurm RUNNING state indicates the allocation is active but does not guarantee server readiness; you must check invocation status and live progress separately to avoid falsely assuming server health.

How do I monitor real-time progress of LLM evaluations on a Slurm cluster?▼

You can monitor real-time progress of LLM evaluations by checking invocation and task status through nemo-evaluator-launcher workflows, which provide terminal-progress tracking and honor benchmark-specific monitoring references.

How do I extract and analyze artifacts from completed LLM evaluation runs?▼

To extract and analyze artifacts from completed LLM evaluation runs, locate the output directories and export logs, then follow benchmark-specific analysis guidance to interpret the resulting metrics accurately.

Can I limit sample sizes and preview resolved configs before starting an eval run?▼

Yes, you can limit samples and preview resolved configurations before starting an evaluation run by using the nemo-evaluator-launcher run workflow to ensure your setup is correct prior to submission.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

launching-evals

Launch and monitor Nemo Evaluator runs across clusters and Slurm jobs.

Official
Advanced
surus-latsurus-lat
8

push-to-latamboard

Merge local benchmark results into the HuggingFace LatamBoard leaderboard dataset.

Official
Advanced
Mekann2904Mekann2904

log-analyzer

Analyze log files to extract error patterns and detect anomalies.

Community
Intermediate
arsityarsity

nemo-evaluator-sdk

Benchmark LLMs across 100+ benchmarks using containerized backends.

Community
Advanced