evaluation

Generate and run LLM evaluation configurations with NeMo Evaluator Launcher.

2.8k|332|Updated Feb 25, 2026

Search Tags:#workflow #llm #benchmarking #evaluation #quantization #nel

Ready to run

npx skills add https://github.com/NVIDIA/skills --skill evaluation-nvidia

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: evaluation
Source: https://github.com/NVIDIA/skills/tree/main/skills/Model-Optimizer/evaluation
Command: npx skills add https://github.com/NVIDIA/skills --skill evaluation-nvidia

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluates the accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). This enables engineers to configure, deploy, and run end-to-end evaluations with production-ready workflows.

Core Features & Use Cases

Workflow automation for model evaluation using NEL.
Production-ready config generation and evaluation execution.
Supports handling workspace setup, model quantization detection, and per-task overrides.
Not intended for quantization (use ptq) or deployment of models.

Quick Start

Run the evaluation workflow by generating a base config and launching Nel to execute the selected benchmarks.

Frequently Asked Questions about evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark LLM accuracy using NeMo Evaluator Launcher?▼

To benchmark LLMs using NeMo Evaluator Launcher, generate a base configuration file and launch NEL to execute end-to-end evaluation workflows on your selected models.

Can I evaluate quantized models with this workflow?▼

Yes, the evaluation workflow supports both quantized and unquantized LLMs. It automatically detects model quantization paths and configures the benchmark execution accordingly for accurate results.

What is the best way to set up a workspace for LLM benchmarking?▼

The best way to set up a workspace for LLM benchmarking is to use the automated workspace management feature, which handles model-path detection and config generation before executing benchmarks.

Does the evaluation workflow handle model quantization and deployment?▼

No, the evaluation workflow does not handle model quantization or deployment. It is strictly designed for configuring and running end-to-end benchmarks, excluding quantization or deployment steps entirely.

How do I apply per-task overrides when running LLM evaluations?▼

To apply per-task overrides when running LLM evaluations, specify the overrides during the automated config generation phase before launching the NeMo Evaluator Launcher to execute the benchmarks.