Standardize scoring metric definitions for AI benchmarks in Benchy YAML configurations.
npx skills add https://github.com/surus-lat/benchy --skill define-scoring
Please help me install this Agent Skill. Skill: define-scoring Source: https://github.com/surus-lat/benchy/tree/main/.agent/skills/define-scoring Command: npx skills add https://github.com/surus-lat/benchy --skill define-scoring
This skill removes the guesswork from evaluating AI performance by providing a structured, standardized way to define how model outputs are graded against expected results.
Ask the assistant to define the scoring section for your current benchmark configuration using the per-field method.
High-intent search queries and answers about installing and using this skill.
You can configure scoring metrics for AI benchmarking by defining scoring types and parameters within a benchmark.yaml configuration file, which standardizes how model outputs are graded against expected results across extraction, classification, and freeform scenarios.
For evaluating data extraction tasks, available scoring methods include per-field extraction scoring with numeric tolerances, binary pass/fail classification, and semantic fuzzy matching to ensure precise and reproducible evaluation logic.
You can generate correct YAML syntax for a benchmark configuration file through guided selection of scoring types and parameters, which maps your evaluation requirements to the framework's internal handler logic for consistent setup.
Yes, the Benchy framework supports binary pass/fail scoring for classification benchmarks by mapping user requirements to internal handler logic, covering classification, extraction, and freeform evaluation scenarios.
Use semantic fuzzy matching instead of per-field scoring when evaluating freeform text outputs rather than structured data extraction, as per-field scoring with numeric tolerances is designed for precise extraction evaluation while fuzzy matching handles semantic similarity.
surus-latGenerate synthetic benchmark datasets as JSONL input-output pairs from task specifications.
surus-latCreate and validate custom AI benchmarking tasks in the Benify framework.
surus-latIntegrate and benchmark Together AI models in Benchy with structured extraction tasks.
surus-latGenerate benchmark task YAML configurations for the Benchy engine.
