Automate end-to-end creation and evaluation of custom LLM benchmarks with BYOB.
npx skills add https://github.com/NVIDIA/skills --skill byob
Please help me install this Agent Skill. Skill: byob Source: https://github.com/NVIDIA/skills/tree/main/skills/NeMo-Evaluator/byob Command: npx skills add https://github.com/NVIDIA/skills --skill byob
BYOB enables researchers and developers to build, customize, and evaluate large language model benchmarks using the BYOB decorator framework, providing reproducible evaluation workflows.
Guide the user through 5 steps to build and evaluate a BYOB benchmark from a dataset.
High-intent search queries and answers about installing and using this skill.
You can build custom LLM benchmarks by using a stepwise workflow that integrates your datasets, prompt templates, and scoring methods to generate reproducible evaluation experiments and reports.
Automating reproducible LLM evaluation workflows involves applying a decorator framework across diverse data sources and prompts to consistently generate benchmarks and reports.
Yes, the LLM evaluation workflow supports dataset handling for both JSONL files and Hugging Face datasets to accommodate diverse data sources.
Yes, you can use LLM-as-Judge support alongside built-in scorers to perform both subjective and objective evaluations when assessing custom benchmarks.
You apply custom prompt templates during benchmark creation by selecting prompt templating options within the stepwise workflow to ensure repeatable experiments across your datasets.
Yes, optional containerization is supported to satisfy integration requirements and ensure consistent environments when running LLM evaluation experiments.
hongmaple0820Initialize Hugging Face dataset repos, stream JSONL updates, and run SQL transformations via DuckDB.
matthew-johnsonBenchmark LLMs across 60+ tasks using YAML-configured harness backends.
sayalinvidiaEvaluate NeMo Evaluator plugins against a running NeMo Platform server.
yo-stevenConvert JSON, CSV, or Hugging Face datasets into BYOB evaluation benchmarks with custom scorers.
