nemo-evaluator-plugin

Name: nemo-evaluator-plugin
Availability: InStock
Author: NVIDIA-NeMo

Execute automated evaluation tasks and metric benchmarking against the NeMo Platform.

58|16|Updated May 14, 2026

Search Tags:#cli #benchmarking #sdk #evaluation #llm-judge #nemo

Ready to run

npx skills add https://github.com/NVIDIA-NeMo/nemo-platform --skill nemo-evaluator-plugin-nvidia-nemo

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-evaluator-plugin
Source: https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/skills/nemo-evaluator-plugin
Command: npx skills add https://github.com/NVIDIA-NeMo/nemo-platform --skill nemo-evaluator-plugin-nvidia-nemo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo_evaluator, nemo_evaluator_sdk, nemo_platform, cloudpickle, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This skill simplifies the complex process of running evaluations against the NeMo Platform, replacing manual API interactions with a structured, repeatable CLI and SDK-based workflow.

Core Features & Use Cases

Evaluation Execution: Run exact-match metrics, LLM-as-judge scoring, and benchmark suites directly from the CLI.
Durable Job Management: Submit long-running evaluation jobs to the cluster and track their status and results asynchronously.
Use Case: A developer needs to validate a new model's performance against a specific dataset; they use this skill to configure an LLM-judge spec and submit it as a durable job to the NeMo Platform.

Quick Start

Use the nemo-evaluator-plugin to run an exact-match evaluation by executing the nemo evaluator evaluate run command with a specified JSON spec file.

Frequently Asked Questions about nemo-evaluator-plugin

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run automated LLM evaluation and benchmarking against the NeMo Platform?▼

You can run automated evaluation tasks and metric benchmarking against the NeMo Platform by using the nemo evaluator evaluate run CLI command with a defined JSON evaluation spec file.

What is the best way to submit long-running evaluation jobs to a remote cluster?▼

The best way to submit long-running evaluation jobs to a remote cluster is by using the NeMo Platform SDK to configure the task and track its status and results asynchronously.

Do I need the NeMo Platform SDK to use LLM-as-judge scoring for model validation?▼

Yes, the NeMo Platform SDK is required to configure and run LLM-as-judge scoring specs for validating model performance against specific datasets.

How does asynchronous job tracking work for NeMo Platform evaluation workflows?▼

Asynchronous job tracking works by submitting durable evaluation jobs to the cluster via the SDK, allowing you to check status and retrieve benchmark results without blocking your local CLI environment.

Can I use a JSON spec file to configure exact-match metrics for local CLI runs?▼

Yes, you can use a JSON spec file to configure exact-match metrics and execute local CLI-based runs directly against the NeMo Platform infrastructure for model performance validation.