nemo-evaluator-sdk

Evaluate LLMs across benchmarks using Docker, Slurm, or cloud backends.

11.5k|842|Updated Nov 3, 2025

Search Tags:#benchmarking #llm evaluation #docker #hpc #nvidia #nemo #reproducible ai

Ready to run

npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill nemo-evaluator-sdk

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-evaluator-sdk
Source: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/11-evaluation/nemo-evaluator
Command: npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill nemo-evaluator-sdk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the complex and time-consuming process of evaluating Large Language Models (LLMs) across a wide array of benchmarks and execution environments.

Core Features & Use Cases

Comprehensive Benchmarking: Evaluates LLMs against 100+ benchmarks from 18+ harnesses (e.g., MMLU, HumanEval, GSM8K, safety, VLM).
Multi-Backend Execution: Supports running evaluations on local Docker, Slurm HPC clusters, or cloud platforms.
Reproducible Evaluation: Utilizes container-first architecture for consistent and reproducible benchmarking.
Use Case: A research team needs to compare the performance of two new LLMs on standard academic benchmarks and safety tests. They can use this Skill to configure and run these evaluations efficiently across their Slurm cluster, generating comparable results.

Quick Start

Use the nemo-evaluator-sdk skill to evaluate the 'meta/llama-3.1-8b-instruct' model on the 'ifeval' task using a local Docker execution.

Frequently Asked Questions about nemo-evaluator-sdk

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run reproducible LLM evaluation across multiple benchmarks?▼

Reproducible LLM evaluation is achieved through a container-first architecture, executing models against over 100 benchmarks from 18+ harnesses. This framework supports multi-backend execution on local Docker, Slurm, or cloud platforms to ensure consistent performance analysis.

Can I evaluate LLMs on a Slurm HPC cluster using Docker?▼

Yes, you can evaluate LLMs on a Slurm HPC cluster using Docker. The framework supports multi-backend execution across local Docker, Slurm HPC clusters, and cloud platforms, allowing scalable benchmarking in diverse high-performance computing environments.

What benchmarks are supported for LLM performance analysis?▼

Supported benchmarks for LLM performance analysis include over 100 benchmarks from 18+ evaluation harnesses. The framework integrates comprehensive academic datasets like MMLU, HumanEval, and GSM8K, alongside specialized safety and VLM evaluations.

How do I compare LLM performance on standard academic and safety tests?▼

Comparing LLM performance on academic and safety tests involves configuring the framework to run evaluations across desired benchmarks. It integrates multiple evaluation harnesses to generate comparable results efficiently across your chosen execution backend.