skills.rest
iani-kuli
iani-kuliCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

generate-synthetic-data

Generate diverse synthetic test tuples for LLM pipeline evaluation.

Updated May 5, 2026
Search Tags:#testing#llm evaluation#data generation#synthetic data#pipeline validation
One-click install
npx skills add https://github.com/iani-kuli/harness_bro --skill generate-synthetic-data-iani-kuli
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: generate-synthetic-data
Source: https://github.com/iani-kuli/harness_bro/tree/main/.claude/skills/curated/evals/generate-synthetic-data
Command: npx skills add https://github.com/iani-kuli/harness_bro --skill generate-synthetic-data-iani-kuli

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill addresses the challenge of evaluating LLM pipelines when real-world user data is scarce, sparse, or insufficient for stress-testing specific failure hypotheses.

Core Features & Use Cases

  • Dimension-Based Generation: Uses a structured tuple-based approach to ensure high-quality, diverse test inputs.
  • Failure Hypothesis Testing: Targets specific failure-prone areas of your pipeline rather than generating generic, low-value data.
  • Use Case: If you are building a real estate assistant and need to test how it handles ambiguous queries from luxury buyers, this skill helps you systematically generate a balanced set of synthetic test cases to validate your pipeline's performance.

Quick Start

Use the generate-synthetic-data skill to create a set of test tuples for my customer support chatbot by defining dimensions for user intent and query complexity.

Frequently Asked Questions about generate-synthetic-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate synthetic data for LLM evaluation when real user traces are unavailable?▼

To generate synthetic data for LLM evaluation, define variation dimensions like user intent and query complexity, then convert those tuples into natural language queries to systematically stress-test your pipeline failure hypotheses.

What is the best way to structure synthetic test inputs for pipeline validation?▼

The best way to structure synthetic test inputs for pipeline validation is using dimension-based tuple generation, which ensures diverse test cases by systematically combining defined variables before converting them into natural language queries.

Can I target specific failure hypotheses when generating synthetic datasets for an LLM assistant?▼

Yes, you can target specific failure hypotheses when generating synthetic datasets by defining precise variation dimensions, allowing you to systematically test failure-prone areas like ambiguous queries instead of generating generic data.

How do I bootstrap an evaluation dataset for a customer support chatbot?▼

To bootstrap an evaluation dataset for a customer support chatbot, define dimensions for user intent and query complexity, then use tuple generation to convert those structural variations into diverse natural language test inputs.

Why use dimension-based tuple generation instead of generic data generation for stress-testing LLMs?▼

Dimension-based tuple generation is preferred over generic data generation for stress-testing LLMs because it systematically targets specific failure-prone areas of your pipeline, ensuring high-quality, diverse test inputs rather than low-value generic cases.

Related Skills

Explore More Data & Analytics Skills→
hamelsmuhamelsmu
1.6k

generate-synthetic-data

Generate diverse synthetic test inputs for LLM pipeline evaluation.

Community
Intermediate
marchattonmarchatton
5

generate-synthetic-data

Generate diverse synthetic test inputs for LLM pipeline evaluation.

Community
Intermediate
zivtechzivtech

test-builder

Generate evaluation suites with fixtures, rubrics, and harness configurations for A/B testing.

Official
Advanced
Avi977Avi977

generate-synthetic-data

Generates dimension-based synthetic inputs to evaluate LLM pipelines via tuples and queries.

Community
Advanced