Launch and monitor Nemo Evaluator runs across clusters and Slurm jobs.
npx skills add https://github.com/NVIDIA/skills --skill launching-evals
Please help me install this Agent Skill. Skill: launching-evals Source: https://github.com/NVIDIA/skills/tree/main/skills/NeMo-Evaluator-Launcher/launching-evals Command: npx skills add https://github.com/NVIDIA/skills --skill launching-evals
Running complex LLM evaluation workflows with Nemo Evaluator Launcher can be error-prone and hard to track across clusters; this skill provides a structured approach to launching, monitoring, debugging, and analyzing evaluations, including artifact export and log inspection.
Submit an evaluation config to start a run and monitor its progress.
High-intent search queries and answers about installing and using this skill.
You can launch and monitor LLM evaluations by submitting a config through the nemo-evaluator-launcher, which orchestrates runs across Slurm clusters and provides live status querying. This skill structures the config submission and status tracking process.
Debug failed Nemo Evaluator runs by inspecting client and server logs retrieved via the nemo-evaluator-launcher. This skill facilitates log inspection and artifact export to pinpoint execution errors across clusters.
Yes, you can export artifacts and analyze benchmarking results from completed Nemo Evaluator jobs. This skill applies benchmark-specific guidance stored under references to analyze metrics across multiple runs.
The best way to track live evaluation progress is by using the nemo-evaluator-launcher to query status and retrieve info. This skill integrates with the launcher to manage live progress monitoring and result collection.
Yes, the Nemo Evaluator Launcher supports submitting custom benchmark configs to start evaluation runs. This skill manages the config submission process to execute and orchestrate the specified LLM evaluations.
yo-stevenRun, monitor, and debug LLM evaluation jobs on Slurm with nemo-evaluator-launcher.
davpatel605-beepEvaluate LLMs against standardized benchmarks like MMLU and GSM8K.
hhhi21gEvaluate large language models across 100+ benchmarks on Docker, Slurm, and cloud.
ovachieverEvaluate language models across 60+ academic benchmarks with lm-evaluation-harness.
