Generate synthetic benchmark datasets as JSONL input-output pairs from task specifications.
npx skills add https://github.com/surus-lat/benchy --skill synthesize-data
Please help me install this Agent Skill. Skill: synthesize-data Source: https://github.com/surus-lat/benchy/tree/main/.agent/skills/synthesize-data Command: npx skills add https://github.com/surus-lat/benchy --skill synthesize-data
This skill solves the cold-start problem for benchmarking by automatically generating synthetic training and evaluation examples when real-world datasets are unavailable.
Ask the assistant to generate 30 examples for your specific benchmark task to populate the training data.
High-intent search queries and answers about installing and using this skill.
Generate synthetic benchmark data by parsing task specifications to produce JSONL files containing input-output pairs. This process creates task-specific data samples to jumpstart evaluation pipelines when real-world datasets are unavailable.
Creating evaluation datasets without real-world samples requires automated generation of synthetic examples. This skill parses your benchmark specifications to generate text and expected output pairs, solving the cold-start problem for new evaluation pipelines.
Benchmark specifications for synthetic data generation must define the input type, output schema, and generation parameters. Providing a valid specification file ensures the generated JSONL samples adhere to your required field schema for structured extraction tasks.
Yes, synthetic data generation supports automated evaluation workflows by creating task-specific data samples for classification benchmarks. It generates input-output pairs based on your specifications to populate training and evaluation pipelines.
Synthetic benchmark data generation includes schema validation to ensure generated data adheres to the required field schema for structured extraction tasks. This guarantees the JSONL output matches your expected data structure for evaluation.
Yes, you need a valid benchmark specification file to generate synthetic JSONL training data. The specification defines the input type, output schema, and parameters required to automatically produce task-specific input-output pairs for your evaluation pipeline.
surus-latGenerate benchmark task YAML configurations for the Benchy engine.
Sumeet138Creates customizable evaluation pipelines for LLM outputs with quantitative and qualitative metrics.
carrot-foundationValidate and standardize JSON Schemas generated from Zod definitions.
JingyiWngGenerate synthetic PDFs with question-answer JSON metadata into Unity Catalog volumes.
