skills.rest
rebeaty
rebeatyCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

benchmark-onboarder

Convert creativity benchmarks into HELM Scenarios with extracted prompts and mapped fields.

1|Updated Jan 10, 2026
Search Tags:#creativity#benchmark#scenario#onboard#prompt-extraction#data-mapping#HELM
One-click install
npx skills add https://github.com/rebeaty/amazon-creativity-benchmark --skill benchmark-onboarder
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark-onboarder
Source: https://github.com/rebeaty/amazon-creativity-benchmark/tree/main/.claude/skills/benchmark-onboarder
Command: npx skills add https://github.com/rebeaty/amazon-creativity-benchmark --skill benchmark-onboarder

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

The Benchmark Onboarder identifies and formalizes creativity benchmarks into HELM-ready Scenarios, enabling consistent evaluation workflows across models and benchmarks.

Core Features & Use Cases

  • Qualifies benchmarks for creativity and evaluability
  • Extracts exact prompts from source papers and maps dataset structures to HELM fields
  • Generates a HELM-compliant scenario.py header with proper metadata and references

Quick Start

Invoke the skill to onboard a benchmark by citing the target paper and providing its dataset, prompts, and evaluation context, so Claude can generate a ready-to-run HELM scenario.

Frequently Asked Questions about benchmark-onboarder

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a dataset into a HELM scenario for creativity evaluation?▼

To convert a dataset into a HELM scenario, extract exact prompts from the source paper, map dataset fields to HELM structures, and generate a scenario.py header with proper metadata and references.

What is the process for onboarding creativity benchmarks into HELM?▼

Onboarding creativity benchmarks into HELM involves qualifying the benchmark for evaluability, extracting exact prompts from source papers, mapping dataset structures to HELM fields, and generating a HELM-compliant scenario.py file.

Can I generate HELM evaluation code directly from a research paper?▼

Yes, you can generate HELM evaluation code from a paper by citing the target paper, providing its dataset and prompts, and documenting evaluation setup via optional annotator or metric notes.

Does onboarding a HELM scenario require exact prompt extraction from the original source?▼

Yes, onboarding a HELM scenario requires extracting exact prompts from source papers to ensure the generated evaluation code accurately reflects the original benchmark's creativity context.

What is the best way to map dataset fields to HELM format for benchmark scenarios?▼

The best way to map dataset fields to HELM format is to qualify the benchmark for creativity, extract exact prompts, and generate a HELM-compliant scenario.py header documenting the evaluation setup.

What limitations exist when qualifying benchmarks for HELM creativity scenarios?▼

Limitations include the requirement to extract exact prompts from source papers and the need to map dataset structures precisely to HELM fields, restricting benchmarks lacking clear evaluation context or structured datasets.

Related Skills

Explore More Software Engineering Skills→
karpathykarpathy
57.1k

read-arxiv-paper

Fetch and unpack arXiv LaTeX sources, then write Markdown summaries.

Community
Basic
tblakex01tblakex01
1

benchmark-models

Benchmark the same prompt across Claude, GPT, and Gemini for latency, tokens, and cost.

Community
Advanced
jaydubya818jaydubya818

advanced-evaluation

Implement LLM-as-a-Judge scoring, pairwise comparison, and bias mitigation.

Community
Advanced
yo-stevenyo-steven

byob

Convert JSON, CSV, or Hugging Face datasets into BYOB evaluation benchmarks with custom scorers.

Community
Advanced