Benchmark Manager

Debugging AILANG benchmarks starts by validating your YAML structure and distinguishing task_prompt from prompt—a common source of 0% pass rates. Use this Skill to inspect the exact prompt your model receives, test with a cheap model first, and trace configuration errors against the Benchmark YAML format and capability schemas (IO, FS, Clock, Net).

In AILANG evaluation benchmarks, task_prompt defines the benchmark task itself, while prompt is what gets sent to the model. Confusing these two causes benchmarks to fail silently. This Skill enforces correct prompt integration to ensure your benchmark configuration matches AILANG's functional requirements.

Creating an AILANG evaluation benchmark requires structuring a YAML file that complies with the Benchmark format, defining task prompts, and validating against capability schemas. This Skill guides you through benchmark creation, validates your YAML for common errors, and provides debugging tools to test your benchmark efficiently.

Yes. This Skill includes automated validation tools that check your YAML files for incorrect prompt usage, schema violations, and format inconsistencies before execution, helping you catch configuration problems early and avoid failed benchmark runs.

AILANG benchmark creation requires ailang, jq, yq, and python3. This Skill assumes these dependencies are installed and uses them to validate YAML structure, parse benchmark configurations, and execute test runs against your evaluation definitions.

A 0% pass rate often signals a prompt configuration problem rather than missing language support. This Skill helps identify and fix the underlying issue—typically incorrect prompt type usage or schema non-compliance—by showing you exactly what prompt the model receives and testing the benchmark with diagnostic tools.