skills.rest
NVIDIA-NeMo
NVIDIA-NeMoOFFICIAL
Explore all skills by this author →
WebsiteGitHub

nemo-evaluator-plugin

Execute automated evaluation tasks and metric benchmarking against the NeMo Platform.

58|16|Updated May 14, 2026
Search Tags:#cli#benchmarking#sdk#evaluation#llm-judge#nemo
One-click install
npx skills add https://github.com/NVIDIA-NeMo/nemo-platform --skill nemo-evaluator-plugin-nvidia-nemo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-evaluator-plugin
Source: https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/skills/nemo-evaluator-plugin
Command: npx skills add https://github.com/NVIDIA-NeMo/nemo-platform --skill nemo-evaluator-plugin-nvidia-nemo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo_evaluator, nemo_evaluator_sdk, nemo_platform, cloudpickle, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This skill simplifies the complex process of running evaluations against the NeMo Platform, replacing manual API interactions with a structured, repeatable CLI and SDK-based workflow.

Core Features & Use Cases

  • Evaluation Execution: Run exact-match metrics, LLM-as-judge scoring, and benchmark suites directly from the CLI.
  • Durable Job Management: Submit long-running evaluation jobs to the cluster and track their status and results asynchronously.
  • Use Case: A developer needs to validate a new model's performance against a specific dataset; they use this skill to configure an LLM-judge spec and submit it as a durable job to the NeMo Platform.

Quick Start

Use the nemo-evaluator-plugin to run an exact-match evaluation by executing the nemo evaluator evaluate run command with a specified JSON spec file.

Frequently Asked Questions about nemo-evaluator-plugin

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run automated LLM evaluation and benchmarking against the NeMo Platform?▼

You can run automated evaluation tasks and metric benchmarking against the NeMo Platform by using the nemo evaluator evaluate run CLI command with a defined JSON evaluation spec file.

What is the best way to submit long-running evaluation jobs to a remote cluster?▼

The best way to submit long-running evaluation jobs to a remote cluster is by using the NeMo Platform SDK to configure the task and track its status and results asynchronously.

Do I need the NeMo Platform SDK to use LLM-as-judge scoring for model validation?▼

Yes, the NeMo Platform SDK is required to configure and run LLM-as-judge scoring specs for validating model performance against specific datasets.

How does asynchronous job tracking work for NeMo Platform evaluation workflows?▼

Asynchronous job tracking works by submitting durable evaluation jobs to the cluster via the SDK, allowing you to check status and retrieve benchmark results without blocking your local CLI environment.

Can I use a JSON spec file to configure exact-match metrics for local CLI runs?▼

Yes, you can use a JSON spec file to configure exact-match metrics and execute local CLI-based runs directly against the NeMo Platform infrastructure for model performance validation.

Related Skills

Explore More Software Engineering Skills→
NVIDIA-NeMoNVIDIA-NeMo
58

evaluator

Automate agent evaluation metric runs and job monitoring in NeMo Platform.

Official
Intermediate
mattwoodcomattwoodco
3

queue

Implement a serverless background job queue with Inngest and PostgreSQL.

Community
Advanced
sayalinvidiasayalinvidia

nemo-evaluator-plugin

Evaluate NeMo Evaluator plugins against a running NeMo Platform server.

Community
Intermediate
KwongFukKwongFuk

palmetto-slurm-workflow

Validate Palmetto Slurm job submissions with cluster guardrails and execution modes.

Community
Advanced