skills.rest
FMurray
FMurrayCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

synthetic-data-generation

Generate realistic synthetic datasets with Faker and Spark and save them to Databricks Unity Catalog Volumes.

Updated Mar 5, 2026
Search Tags:#data generation#test data#synthetic data#faker#spark#databricks#unity catalog
One-click install
npx skills add https://github.com/FMurray/mlfts --skill synthetic-data-generation-fmurray
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: synthetic-data-generation
Source: https://github.com/FMurray/mlfts/tree/main/.agents/skills/synthetic-data-generation
Command: npx skills add https://github.com/FMurray/mlfts --skill synthetic-data-generation-fmurray

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires faker, holidays, numpy, pandas, pyspark.

What problem does it solve? Creating believable test and demo datasets for Databricks is tedious: naive random data lacks realistic distributions, referential integrity, and time-based patterns, making downstream pipelines and dashboards unconvincing. This Skill guides the generation of story-driven synthetic data with correlated attributes and saves it as raw Parquet files to Unity Catalog Volumes. ## Core Features & Use Cases - Realistic Distributions: Uses log-normal, exponential, Pareto, and weighted categorical distributions instead of uniform randomness, plus weekday/weekend, holiday, seasonality, and incident-spike time patterns. - Referential Integrity & Row Coherence: Generates master tables first, then child tables with valid foreign keys, and correlates attributes within rows (tier affects priority, priority affects resolution time, resolution affects CSAT). - Databricks-Native Workflow: Writes Python scripts locally, executes them on Databricks via MCP tools with cluster/context reuse, creates catalog/schema/volume infrastructure in-script, and saves raw Parquet to Volumes for downstream Spark Declarative Pipelines. - Use Case: Generate 2,500 customers, 25,000 orders, and 8,000 support tickets spanning the last 6 months with an Auth incident spike three weeks ago, then feed the raw data into a bronze/silver/gold medallion pipeline. ## Quick Start Ask the AI to generate a synthetic dataset of customers, orders, and support tickets for the last six months and save it as Parquet files to a Unity Catalog Volume in the ai_dev_kit catalog.

Frequently Asked Questions about synthetic-data-generation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate synthetic data for Databricks with Python?▼

Write a Python script locally using Faker, NumPy, and pandas to build DataFrames, then execute it on Databricks with the run_python_file_on_databricks MCP tool. Convert the pandas DataFrames to Spark and save them as Parquet files to a Unity Catalog Volume.

How do I create realistic distributions in synthetic data?▼

Use log-normal distributions for prices and amounts, exponential for durations and resolution times, Pareto for popularity, and weighted categorical sampling for segments. Avoid uniform distributions, and add time-based multipliers for weekends, holidays, seasonality, and event spikes.

Should synthetic data be saved as tables or files in Databricks?▼

Save generated data as raw Parquet files to a Unity Catalog Volume, not as tables, unless explicitly requested. This keeps the data as a raw source for a downstream Spark Declarative Pipeline that handles bronze, silver, and gold layer transformations.

How do I maintain referential integrity across synthetic tables?▼

Generate master tables like customers first, then build lookup dictionaries of their IDs and attributes. Sample from those IDs when creating child tables such as orders and tickets so every foreign key references a valid parent row.

Why does my Databricks script execution fail with import errors?▼

Libraries like faker and holidays are not pre-installed on Databricks clusters. Install them with the execute_databricks_command tool using %pip install, then reuse the same cluster_id and context_id so the installed libraries persist across executions.

How many rows should synthetic data have for aggregation?▼

Generate at least 10K-50K total rows so patterns survive GROUP BY aggregation in downstream pipelines. Aim for 50-100 records per day for time series, 500+ per category, and 5-20 events per customer for meaningful analysis.

Related Skills

Explore More Data & Analytics Skills→
JingyiWngJingyiWng

databricks-genie

Create and query Databricks Genie Spaces via natural language.

Community
Advanced
databrickslabsdatabrickslabs
38

databricks-spark-declarative-pipelines

Automate creation, configuration, and optimization of Databricks Lakeflow Spark Declarative Pipelines.

Official
Advanced
modus-bimodus-bi

create-dashboard

Create BI dashboards from natural-language prompts with SQL datasets and visualizations.

Official
Advanced
proffesor-for-testingproffesor-for-testing
436

test-data-management

Generate privacy-preserving synthetic test data with Faker and masking.

Community
Intermediate