skills.rest
adamatdevops
adamatdevopsCOMMUNITY
Explore all skills by this author →
GitHub

generate-synthetic-data

Generate synthetic evaluation inputs covering LLM pipeline failure spaces.

1|Updated Jan 7, 2026
Search Tags:#llm evaluation#synthetic data#test case generation#quality filtering#natural language queries#trace collection#dataset coverage
One-click install
npx skills add https://github.com/adamatdevops/forge-works --skill generate-synthetic-data-adamatdevops
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: generate-synthetic-data
Source: https://github.com/adamatdevops/forge-works/tree/main/.skills/hamelsmu/generate-synthetic-data
Command: npx skills add https://github.com/adamatdevops/forge-works --skill generate-synthetic-data-adamatdevops

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It helps you create diverse, realistic synthetic inputs that expose likely failure modes in an LLM pipeline, so you can evaluate quality and robustness without relying only on scarce real user traffic.

Core Features & Use Cases

  • Dimension-driven coverage planning: choose failure-oriented axes of variation (dimensions) rather than arbitrary randomness.
  • Tuple-to-query generation workflow: draft tuples with user confirmation, then generate additional tuples, convert them into realistic natural-language queries, and filter for quality.
  • Trace-first evaluation execution: run the resulting queries through your full LLM pipeline to collect complete traces for ~100 high-quality, diverse test cases.

Use case:

Generating a comprehensive set of search/chat queries for an intent-classification or retrieval pipeline, ensuring ambiguous, out-of-scope, and constraint-heavy scenarios are represented so regressions are caught early.

Quick Start

Use the skill to define 3+ failure-targeted dimensions, confirm 20 realistic tuples, generate additional tuples and natural-language queries, filter them for realism, and run them through your LLM pipeline while capturing full traces.

Frequently Asked Questions about generate-synthetic-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate synthetic test cases for LLM evaluation when I lack real user traffic?▼

Generate synthetic data for LLM evaluation by defining dimension-guided tuples, converting them into natural-language queries, and applying realism filtering to produce roughly 100 diverse, high-quality test traces for your pipeline.

What is dimension-driven coverage planning for synthetic data generation?▼

Dimension-driven coverage planning involves selecting failure-oriented axes of variation to guide synthetic data generation, ensuring your LLM evaluation covers ambiguous, out-of-scope, and constraint-heavy scenarios instead of relying on arbitrary randomness.

How do I bootstrap evaluation datasets for intent classification and retrieval pipelines?▼

Bootstrap evaluation datasets for intent classification by defining failure dimensions, confirming tuples, generating natural-language queries, filtering for realism, and running them through your pipeline to capture full execution traces.

Can I use synthetic data generation to catch regressions in chat and search pipelines?▼

Yes, synthetic data generation catches regressions in chat and search pipelines by generating diverse, constraint-heavy queries that saturate the failure space, capturing full execution traces to expose quality and robustness issues early.

How does tuple-to-query generation work for creating realistic natural-language queries?▼

Tuple-to-query generation works via a two-step LLM process: first draft and confirm dimension-guided tuples, then use an LLM to convert those tuples into realistic natural-language queries, which are subsequently filtered for quality and realism.

What are the limitations of using synthetic data for LLM quality filtering and trace collection?▼

A limitation of synthetic data generation is that output quality depends entirely on your initially defined failure dimensions and the effectiveness of the realism filtering step, requiring user-confirmed tuples and two-step LLM generation to achieve roughly 100 diverse traces.

Related Skills

Explore More Data & Analytics Skills→
marchattonmarchatton
5

generate-synthetic-data

Generate diverse synthetic test inputs for LLM pipeline evaluation.

Community
Intermediate
hamelsmuhamelsmu
1.6k

generate-synthetic-data

Generate diverse synthetic test inputs for LLM pipeline evaluation.

Community
Intermediate
patronus-aipatronus-ai
2

llm-evaluation

Compute automated metrics and aggregate scores for LLM evaluation suites.

Official
Advanced
Avi977Avi977

generate-synthetic-data

Generates dimension-based synthetic inputs to evaluate LLM pipelines via tuples and queries.

Community
Advanced