skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-evaluator-plugin

Evaluate NeMo Evaluator plugins against a running NeMo Platform server.

Updated Apr 16, 2026
Search Tags:#debugging#evaluation#plugin#spec#llm-judge#evaluator#nemo-platform
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-evaluator-plugin
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-evaluator-plugin
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-evaluator-plugin
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-evaluator-plugin

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo_evaluator, nemo_evaluator_sdk, nemo_platform, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

Provides guidance for evaluating NeMo Evaluator plugins against a running NeMo Platform server, including CLI usage, metric types, evaluation specs, and workflow best practices.

Core Features & Use Cases

  • Detailed instructions for running evaluation workflows with nemo evaluator, supporting metric types such as exact-match and llm-judge, and guidance on creating durable evaluation jobs.
  • References to API authentication, troubleshooting, and examples in the assets to help teams implement reliable evaluation pipelines.

Quick Start

To begin, load or generate an evaluation spec and run a Nemo Evaluator workflow against a local NeMo Platform server.

Frequently Asked Questions about nemo-evaluator-plugin

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate NeMo Evaluator plugins against a running NeMo Platform server?▼

Run NeMo Evaluator plugin evaluation by loading or generating an evaluation spec and executing the workflow via CLI against your running NeMo Platform server. This covers metric types, evaluation specs, and troubleshooting for local and durable deployments.

What metric types are supported by the NeMo Evaluator for evaluating LLM outputs?▼

NeMo Evaluator supports metric types such as exact-match and llm-judge for evaluating outputs. These metrics are defined within your evaluation spec to ensure reliable and auditable assessment of model performance.

Can I create durable evaluation jobs using the NeMo Evaluator CLI?▼

Yes, you can create durable evaluation jobs using the NeMo Evaluator CLI. The tooling provides detailed instructions and workflow best practices for configuring and running these durable evaluation pipelines on a NeMo Platform server.

Do I need API authentication configured to run NeMo Evaluator workflows?▼

Yes, API authentication is required to securely run NeMo Evaluator workflows against a NeMo Platform server. The evaluation guidance explicitly references API auth setup to ensure safe and auditable execution of your evaluation jobs.

Why is my NeMo Evaluator plugin evaluation failing on a local deployment?▼

NeMo Evaluator plugin evaluation failures on local deployments often stem from incorrect evaluation specs or API authentication issues. The Skill provides troubleshooting references and example specs to help resolve these local and durable deployment errors.

Related Skills

Explore More Software Engineering Skills→
NVIDIA-NeMoNVIDIA-NeMo
58

nemo-files

Manage filesets and upload data within the NeMo platform using the nemo CLI.

Official
Basic
coval-aicoval-ai
2

review-llm-annotations-and-improve-prompt

Analyze human versus LLM judge annotation discrepancies to refine evaluation prompts.

Official
Advanced
thesammykinsthesammykins

llm-rag-engineer

Build RAG pipelines with hybrid search, re-ranking, and evaluation.

Community
Advanced
NVIDIA-NeMoNVIDIA-NeMo
58

creating-a-plugin

Create and register modular plugins for the NeMo Platform.

Official
Advanced