Generate benchmark task YAML configurations for the Benchy engine.
npx skills add https://github.com/surus-lat/benchy --skill define-task
Please help me install this Agent Skill. Skill: define-task Source: https://github.com/surus-lat/benchy/tree/main/.agent/skills/define-task Command: npx skills add https://github.com/surus-lat/benchy --skill define-task
This skill removes the complexity of configuring AI evaluation benchmarks by guiding users through a structured process to define their AI's specific task requirements.
Ask the assistant to define a new benchmark task for your AI system by describing what it does and what inputs and outputs it handles.
High-intent search queries and answers about installing and using this skill.
To define AI benchmark tasks, you describe your system's requirements and the skill generates a compliant YAML specification with input types and output schemas for the Benchy benchmarking engine.
You can configure benchmark tasks for extraction, classification, question-answering, and translation workflows, standardizing task categories without needing knowledge of internal handler classes.
Yes, generating benchmark YAML configurations is possible without coding handler classes, as the skill guides you through a structured process to decouple task definition from internal implementation details.
The best way to structure input types and output schemas for an AI evaluation is through a standardized YAML task definition, which automatically captures your field requirements for the benchmark.
No dependencies are required to create benchmark configurations, as the skill operates independently to guide task definition and output the necessary YAML file for your AI evaluation framework.
You need a standardized YAML task definition when you want to remove configuration complexity and integrate specific AI task requirements, such as invoice data extraction, into the Benchy benchmarking engine.
JigsawStackConvert prompts into schema-constrained JSON objects using Zod or Pydantic.
surus-latStandardize scoring metric definitions for AI benchmarks in Benchy YAML configurations.
dawsonblockRun lm-evaluation-harness benchmarks across 60+ tasks and export metrics.
surus-latGenerate synthetic benchmark datasets as JSONL input-output pairs from task specifications.
