nemo-evaluator-sdk

Reproducible LLM evaluation is achieved through a container-first architecture, executing models against over 100 benchmarks from 18+ harnesses. This framework supports multi-backend execution on local Docker, Slurm, or cloud platforms to ensure consistent performance analysis.

Yes, you can evaluate LLMs on a Slurm HPC cluster using Docker. The framework supports multi-backend execution across local Docker, Slurm HPC clusters, and cloud platforms, allowing scalable benchmarking in diverse high-performance computing environments.

Supported benchmarks for LLM performance analysis include over 100 benchmarks from 18+ evaluation harnesses. The framework integrates comprehensive academic datasets like MMLU, HumanEval, and GSM8K, alongside specialized safety and VLM evaluations.

Comparing LLM performance on academic and safety tests involves configuring the framework to run evaluations across desired benchmarks. It integrates multiple evaluation harnesses to generate comparable results efficiently across your chosen execution backend.