nemo-evaluator-sdk

What problem does it solve?

This Skill streamlines the complex and time-consuming process of evaluating Large Language Models (LLMs) across a wide array of benchmarks and execution environments.

Core Features & Use Cases

• Comprehensive Benchmarking: Evaluates LLMs against 100+ benchmarks from 18+ harnesses (e.g., MMLU, HumanEval, GSM8K, safety, VLM).
• Multi-Backend Execution: Supports running evaluations on local Docker, Slurm HPC clusters, or cloud platforms.
• Reproducible Evaluation: Utilizes container-first architecture for consistent and reproducible benchmarking.
• Use Case: A research team needs to compare the performance of two new LLMs on standard academic benchmarks and safety tests. They can use this Skill to configure and run these evaluations efficiently across their Slurm cluster, generating comparable results.

Quick Start

Use the nemo-evaluator-sdk skill to evaluate the 'meta/llama-3.1-8b-instruct' model on the 'ifeval' task using a local Docker execution.