databricks-synthetic-data-gen

Generate realistic synthetic datasets with Spark and Faker for Databricks.

Updated Mar 25, 2026

Search Tags:#generate #faker #spark #databricks #synthetic-data #pandas-udf

Ready to run

npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-synthetic-data-gen-jingyiwng

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher/tree/main/.claude/skills/databricks-synthetic-data-gen
Command: npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-synthetic-data-gen-jingyiwng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, pandas, faker, databricks-connect, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Synthetic datasets are often needed for testing, demos, model training, and analytics but are hard to produce with realistic distributions, referential integrity, and at production scale. This Skill centralizes patterns, tooling, and safe execution guidance to produce coherent story-driven datasets directly into Databricks storage or Unity Catalog.

Core Features & Use Cases

Spark + Faker + Pandas UDFs for scalable, parallel generation across thousands to millions of rows.
Serverless-first execution with fallbacks for classic clusters or local Polars generation for very small datasets.
Unity Catalog integration & safety: prompts for catalog/schema, creates schema/volume if missing, enforces referential integrity, and avoids cache/persist on serverless compute.
Outputs & examples: write to Parquet/JSON/CSV/Delta, add table/column comments, and generate domain scenarios (e-commerce, IoT, CRM, finance) with realistic distributions.

Quick Start

Generate a synthetic e-commerce dataset of 10,000 customers and 50,000 orders into your specified catalog.schema on serverless compute and write to the raw_data volume as Parquet.

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate realistic synthetic data at scale in Databricks?▼

Generate realistic synthetic data at scale in Databricks by applying Spark and Faker with Pandas UDFs to produce millions of rows across domains like e-commerce and IoT, writing outputs directly to Parquet, JSON, CSV, or Delta formats.

What is the best way to create synthetic datasets with referential integrity in Unity Catalog?▼

Creating synthetic datasets with referential integrity in Unity Catalog involves providing your catalog and schema, letting the generation process create missing schemas or volumes, and enforcing relational coherence across story-driven domain tables.

Can I use Faker with Spark Pandas UDFs for synthetic data generation on serverless compute?▼

Faker works with Spark Pandas UDFs on serverless compute to parallelize synthetic data generation across thousands to millions of rows, intentionally avoiding cache and persist operations to maintain serverless compatibility.

How do I output synthetic e-commerce and IoT datasets to Delta format in Databricks?▼

Output synthetic e-commerce and IoT datasets to Delta format in Databricks by configuring Spark and Faker generation jobs to write story-driven, realistically distributed records directly into specified Unity Catalog volumes.

Does synthetic data generation on Databricks serverless compute have limitations with caching?▼

Synthetic data generation on Databricks serverless compute has limitations requiring the avoidance of cache and persist operations, with fallback options available for classic clusters or local generation for very small datasets.