Validate YAML benchmark specification files for completeness and structural integrity.
npx skills add https://github.com/surus-lat/benchy --skill validate-surus-lat
Please help me install this Agent Skill. Skill: validate Source: https://github.com/surus-lat/benchy/tree/main/.agent/skills/validate Command: npx skills add https://github.com/surus-lat/benchy --skill validate-surus-lat
This Skill resolves configuration errors in benchmark specifications, preventing failed runs and saving time by identifying issues before the evaluation engine starts.
Run the validate skill to check the benchmark configuration file located at benchmarks/my-benchmark.yaml for any errors.
High-intent search queries and answers about installing and using this skill.
To validate a benchmark specification, run a validation check against your YAML file to ensure all task, scoring, and model parameters are correctly configured. This process scans for missing fields, invalid types, or broken paths to prevent failed runs.
Benchmark configuration failures often stem from incomplete or structurally invalid YAML definitions. Running a pre-execution validation identifies configuration errors like missing fields or broken paths and provides diagnostic feedback mapping to specific remediation steps.
The best way to check benchmark YAML files for missing fields is to run automated validation. It scans the benchmark definitions for structural integrity, detects invalid types or broken paths, and translates technical errors into actionable advice.
Yes, you can get plain English diagnostics for benchmark configuration errors. The validation process translates complex technical errors into actionable advice, pointing directly to the specific configuration skill required to fix the issue within the workflow.
You need to run benchmark specification validation after defining a new benchmark task and configuring the model, but before initiating a costly or time-consuming evaluation. This ensures all parameters are correctly set and prevents failed runs.
Yes, validation operates directly on YAML-based benchmark definitions. It verifies the completeness and structural integrity of these files by ensuring all task, scoring, and model parameters are correctly configured before the evaluation engine starts.
ovachieverEvaluate language models across 60+ academic benchmarks with lm-evaluation-harness.
moltis-orgEvaluate LLM performance across 60+ academic benchmarks with multiple backends.
alexajunoGenerate interactive HTML playgrounds with visual controls and natural language prompts.
bitwisecookFix Tcl iRules by iteratively applying LSP diagnostics and automated corrections.
