byob

Convert JSON, CSV, or Hugging Face datasets into BYOB evaluation benchmarks with custom scorers.

Updated May 23, 2026

Search Tags:#llm-as-judge #benchmarking #llm evaluation #scorers #JSONL #prompt templating #dockerization

Ready to run

npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill byob-yo-steven

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: byob
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/NeMo-Evaluator/byob
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill byob-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

BYOB helps you stop relying on one-size-fits-all evaluation by guiding you to create benchmarks tailored to your dataset, prompt style, and scoring rules.

Core Features & Use Cases

Custom benchmark design: Create a new evaluation benchmark from your dataset using the BYOB decorator framework.
Dataset-to-eval workflow: Validate your JSONL schema, map fields, and generate a prompt template with {field} placeholders.
Scoring options: Use built-in scorers or generate a custom scorer, with an enforced scorer smoke test before compilation.
Advanced eval modes: Support LLM-as-Judge (subjective grading), eval-only scoring from pre-generated responses, and multiple-choice loglikelihood scoring.
Packaging and execution: Compile a benchmark with the CLI, optionally containerize it, and run it to produce results.

Quick Start

Ask for help creating a BYOB benchmark from your dataset and selecting a scoring method, then compile and run it with the BYOB CLI.

Frequently Asked Questions about byob

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create a custom LLM evaluation benchmark from my own dataset?▼

To create a custom LLM evaluation benchmark, you convert your JSON or CSV dataset into a BYOB benchmark using a validated JSONL schema, map fields to a prompt template, and apply a compatible scorer for compilation.

Can I use LLM-as-Judge for subjective grading in my benchmark evaluation?▼

Yes, LLM-as-Judge is supported for subjective grading in custom benchmark evaluation. You can also use deterministic scoring, pre-generated response eval-only scoring, and multiple-choice loglikelihood evaluation modes.

How do I generate a prompt template for benchmarking from JSONL fields?▼

You generate a prompt template for benchmarking by mapping your validated JSONL dataset fields directly into the template using {field} placeholders, ensuring dataset variables align with prompt inputs before compilation.

What's the best way to validate a custom scorer before compiling a benchmark?▼

The best way to validate a custom scorer is through an enforced scorer smoke test. You must run this smoke test before compilation to ensure your scorer correctly returns dict metrics for the benchmark.

Does BYOB support multiple-choice logprob evaluation for LLMs?▼

Yes, multiple-choice logprob evaluation is fully supported. It provides loglikelihood scoring for multiple-choice questions alongside standard deterministic and LLM-as-Judge subjective grading options.

Why do I need a validated JSONL schema for custom benchmark creation?▼

A validated JSONL schema is required for custom benchmark creation to ensure dataset fields correctly map to prompt template placeholders and scorer inputs, preventing structural errors during CLI compilation and execution.