generate-synthetic-data

Generate diverse synthetic test inputs for LLM pipeline evaluation.

1.6k|164|Updated Mar 1, 2026

Search Tags:#prompt engineering #llm evaluation #synthetic data #test data generation #data augmentation #data diversity

Ready to run

npx skills add https://github.com/hamelsmu/evals-skills --skill generate-synthetic-data-hamelsmu

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: generate-synthetic-data
Source: https://github.com/hamelsmu/evals-skills/tree/main/skills/generate-synthetic-data
Command: npx skills add https://github.com/hamelsmu/evals-skills --skill generate-synthetic-data-hamelsmu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill generates diverse, realistic test inputs for LLM pipelines when real user data is sparse or to stress-test specific failure hypotheses.

Core Features & Use Cases

Dimension-Based Generation: Creates synthetic data by combining predefined dimensions (axes of variation) and their values.
Iterative Refinement: Guides users through defining dimensions, drafting tuples, and converting them into natural language queries.
Use Case: Bootstrapping an evaluation dataset for a new customer support chatbot by generating varied user queries covering different intents and user personas.

Quick Start

Use the generate-synthetic-data skill to create 100 diverse traces for your LLM pipeline.

Frequently Asked Questions about generate-synthetic-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate synthetic test data for LLM evaluation when real user data is sparse?▼

You can generate synthetic test data by defining dimensions of variation, drafting tuples, using an LLM to generate additional tuples, and converting them into natural language queries to yield roughly 100 high-quality traces for pipeline evaluation.

What is dimension-based tuple generation for data augmentation?▼

Dimension-based tuple generation creates data augmentation by combining predefined axes of variation and their values to form tuples, which are then converted into diverse natural language queries for testing LLM pipelines.

Can I use synthetic data generation to stress-test specific failure hypotheses in my LLM pipeline?▼

Yes, generating synthetic data allows you to stress-test specific failure hypotheses by creating targeted, dimension-based test inputs that address scenarios where real user data is sparse or insufficient.

How many synthetic traces can I expect from this test data generation process?▼

The test data generation process produces approximately 100 high-quality traces through an iterative process of dimension definition, tuple generation, natural language query conversion, and quality filtering.

What's the best way to bootstrap an evaluation dataset for a new customer support chatbot?▼

The best way to bootstrap an evaluation dataset is generating diverse synthetic test inputs through dimension-based tuple generation, covering different user intents and personas to stress-test the LLM pipeline.

Does generating synthetic test inputs require any external dependencies or libraries?▼

Generating synthetic test inputs requires no external dependencies, utilizing a multi-step internal process involving dimension definition, tuple drafting, LLM-based generation, and quality filtering to produce evaluation traces.