nemo-evaluator-sdk

Run reproducible LLM benchmarks by using containerized backends that support local Docker, Slurm HPC, and cloud environments to execute scalable evaluations across varied hardware. Configure benchmarks via a CLI or Python API to generate structured results.

Yes, you can evaluate models on MMLU and GSM8K using Slurm HPC. The multi-backend evaluation supports Slurm for parallel benchmarking across 100+ benchmarks, including safety and VLM domains, yielding structured results exports.

Containerized benchmarking works by packaging LLM evaluation harnesses into Docker backends, ensuring reproducible results. It exposes a CLI and Python API to configure 100+ benchmarks and export structured outputs for CI/CD integration.

Integrate LLM benchmarking into research pipelines by utilizing the exposed Python API for enterprise-grade configuration. This allows automated execution across local Docker or cloud backends, exporting structured results from 100+ benchmark harnesses.

Docker is required to run containerized backends for reproducible results locally. However, if Docker is unavailable, you can configure the evaluation to run on Slurm HPC or cloud backends instead to achieve scalable parallel benchmarking.

A multi-backend approach enables scalable evaluations across varied hardware configurations. Supporting local Docker, Slurm HPC, and cloud backends allows you to parallelize benchmarking across 100+ benchmarks while maintaining reproducible results.