Generate and run LLM evaluation configurations with NeMo Evaluator Launcher.
npx skills add https://github.com/NVIDIA/skills --skill evaluation-nvidia
Please help me install this Agent Skill. Skill: evaluation Source: https://github.com/NVIDIA/skills/tree/main/skills/Model-Optimizer/evaluation Command: npx skills add https://github.com/NVIDIA/skills --skill evaluation-nvidia
Evaluates the accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). This enables engineers to configure, deploy, and run end-to-end evaluations with production-ready workflows.
Run the evaluation workflow by generating a base config and launching Nel to execute the selected benchmarks.
High-intent search queries and answers about installing and using this skill.
To benchmark LLMs using NeMo Evaluator Launcher, generate a base configuration file and launch NEL to execute end-to-end evaluation workflows on your selected models.
Yes, the evaluation workflow supports both quantized and unquantized LLMs. It automatically detects model quantization paths and configures the benchmark execution accordingly for accurate results.
The best way to set up a workspace for LLM benchmarking is to use the automated workspace management feature, which handles model-path detection and config generation before executing benchmarks.
No, the evaluation workflow does not handle model quantization or deployment. It is strictly designed for configuring and running end-to-end benchmarks, excluding quantization or deployment steps entirely.
To apply per-task overrides when running LLM evaluations, specify the overrides during the automated config generation phase before launching the NeMo Evaluator Launcher to execute the benchmarks.
yo-stevenGenerate NEL configurations and execute LLM accuracy benchmarks on local or SLURM backends.
KrishpotanwarBenchmark, validate, and optimize Codex v3 model performance across speed, memory, and stability.
ovachieverCompress large language models to 4-bit precision with GPTQ.
AndyAnh174Evaluates LLM-powered systems using automated metrics, human feedback, and benchmarking.
