skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

evaluation

Generate NEL configurations and execute LLM accuracy benchmarks on local or SLURM backends.

Updated May 23, 2026
Search Tags:#benchmarking#llm evaluation#model quantization#nemo evaluator launcher#nel config generation#slurm execution#mlflow wandb
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill evaluation-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluation
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Model-Optimizer/evaluation
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill evaluation-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the trial-and-error involved in evaluating quantized and unquantized LLMs by generating correct NeMo Evaluator Launcher (NEL) configurations and guiding you through running evaluations reliably.

Core Features & Use Cases

  • Config generation for evaluation: Builds a production-ready NEL YAML workflow for accuracy and benchmark runs, including model path/handle and required placeholders.
  • Quantization-aware setup: Detects ModelOpt quantization from checkpoint metadata and applies the correct vLLM/SGLang flags to match the quantized format.
  • Model-card driven parameterization: Uses web research to extract sampling, context length, and deployment-specific requirements, then maps them into NEL config fields.
  • Benchmark task orchestration: Supports common evaluation sets (e.g., MMLU, GPQA, AIME, code and math benchmarks) via task snippets, then iteratively confirms the final task list.

Quick Start

Tell the AI: "Build an SLURM vLLM evaluation config for my FP8 ModelOpt checkpoint at ./my-fp8-model and run MMLU-Pro and GPQA Diamond, then help me fill in the remaining SLURM placeholders."

Frequently Asked Questions about evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate a NeMo Evaluator Launcher config for an LLM accuracy benchmark?▼

To generate a NeMo Evaluator Launcher config for an LLM accuracy benchmark, you can use this Skill to build a production-ready NEL YAML workflow, populating model paths and required placeholders for tasks like MMLU and GPQA.

How do I run evaluations on a quantized LLM using vLLM?▼

Running evaluations on a quantized LLM using vLLM requires applying the correct format flags. This Skill detects ModelOpt quantization from checkpoint metadata and automatically applies the necessary vLLM or SGLang flags for the evaluation run.

Can I execute LLM benchmark evaluations on a SLURM cluster?▼

Yes, you can execute LLM benchmark evaluations on a SLURM cluster. This Skill configures and runs accuracy and benchmark suites across local or SLURM execution backends, allowing you to run dry-run, test, or full evaluations with monitoring guidance.

What benchmark tasks are supported for LLM evaluation?▼

Supported benchmark tasks for LLM evaluation include common sets such as MMLU, GPQA, AIME, alongside various code, math, and safety suites. The Skill orchestrates these tasks via snippets and iteratively confirms the final task list.

How do I export LLM evaluation results to MLflow or wandb?▼

To export LLM evaluation results to MLflow or wandb, you can utilize this Skill's optional export functionality. It configures the NEL workflow to automatically track and export benchmark and accuracy metrics to your preferred platform.

How do I resolve missing placeholder fields in an NEL config?▼

To resolve missing placeholder fields in an NEL config, this Skill uses web research to extract sampling, context length, and deployment-specific requirements from model cards, automatically mapping them into the appropriate NEL configuration fields.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

evaluation

Generate and run LLM evaluation configurations with NeMo Evaluator Launcher.

Official
Advanced
amdamd
154

quark-torch-llm-eval

Evaluate LLM accuracy on AMD ROCm with vLLM, SGLang, or ATOM serving.

Official
Advanced
annealing-inversionannealing-inversion

sglang-diffusion-modelopt-quant

Quantize diffusion DiT models with NVIDIA ModelOpt for SGLang Diffusion.

Community
Advanced
yo-stevenyo-steven

nel-assistant

Generate NEL YAML evaluation configs interactively across execution backends.

Community
Advanced