byob

Automate end-to-end creation and evaluation of custom LLM benchmarks with BYOB.

2.8k|332|Updated Feb 25, 2026

Search Tags:#automation #evaluation #benchmark #llm-evaluation #scorer #nemo-evaluator #byob

Ready to run

npx skills add https://github.com/NVIDIA/skills --skill byob

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: byob
Source: https://github.com/NVIDIA/skills/tree/main/skills/NeMo-Evaluator/byob
Command: npx skills add https://github.com/NVIDIA/skills --skill byob

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

BYOB enables researchers and developers to build, customize, and evaluate large language model benchmarks using the BYOB decorator framework, providing reproducible evaluation workflows.

Core Features & Use Cases

Stepwise workflow guiding users through 5 steps to construct and assess bespoke benchmarks.
BYOB API integration with datasets, prompts, and scoring methods, enabling repeatable experiments and reporting.
LLM-as-Judge support and built-in scorers for objective and subjective evaluation.

Quick Start

Guide the user through 5 steps to build and evaluate a BYOB benchmark from a dataset.

Frequently Asked Questions about byob

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build and evaluate custom LLM benchmarks from my own datasets?▼

You can build custom LLM benchmarks by using a stepwise workflow that integrates your datasets, prompt templates, and scoring methods to generate reproducible evaluation experiments and reports.

What is the best way to automate reproducible LLM evaluation workflows?▼

Automating reproducible LLM evaluation workflows involves applying a decorator framework across diverse data sources and prompts to consistently generate benchmarks and reports.

Does this LLM evaluation workflow support JSONL and Hugging Face datasets?▼

Yes, the LLM evaluation workflow supports dataset handling for both JSONL files and Hugging Face datasets to accommodate diverse data sources.

Can I use LLM-as-Judge for subjective evaluation in custom benchmarks?▼

Yes, you can use LLM-as-Judge support alongside built-in scorers to perform both subjective and objective evaluations when assessing custom benchmarks.

How do I apply custom prompt templates during LLM benchmark creation?▼

You apply custom prompt templates during benchmark creation by selecting prompt templating options within the stepwise workflow to ensure repeatable experiments across your datasets.

Are there options for containerization when running LLM evaluation experiments?▼

Yes, optional containerization is supported to satisfy integration requirements and ensure consistent environments when running LLM evaluation experiments.