skills.rest
JingyiWng
JingyiWngCOMMUNITY
Explore all skills by this author →
GitHub

databricks-synthetic-data-gen

Generate realistic synthetic datasets with Spark and Faker for Databricks.

Updated Mar 25, 2026
Search Tags:#generate#faker#spark#databricks#synthetic-data#pandas-udf
One-click install
npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-synthetic-data-gen-jingyiwng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher/tree/main/.claude/skills/databricks-synthetic-data-gen
Command: npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-synthetic-data-gen-jingyiwng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, pandas, faker, databricks-connect, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Synthetic datasets are often needed for testing, demos, model training, and analytics but are hard to produce with realistic distributions, referential integrity, and at production scale. This Skill centralizes patterns, tooling, and safe execution guidance to produce coherent story-driven datasets directly into Databricks storage or Unity Catalog.

Core Features & Use Cases

  • Spark + Faker + Pandas UDFs for scalable, parallel generation across thousands to millions of rows.
  • Serverless-first execution with fallbacks for classic clusters or local Polars generation for very small datasets.
  • Unity Catalog integration & safety: prompts for catalog/schema, creates schema/volume if missing, enforces referential integrity, and avoids cache/persist on serverless compute.
  • Outputs & examples: write to Parquet/JSON/CSV/Delta, add table/column comments, and generate domain scenarios (e-commerce, IoT, CRM, finance) with realistic distributions.

Quick Start

Generate a synthetic e-commerce dataset of 10,000 customers and 50,000 orders into your specified catalog.schema on serverless compute and write to the raw_data volume as Parquet.

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate realistic synthetic data at scale in Databricks?▼

Generate realistic synthetic data at scale in Databricks by applying Spark and Faker with Pandas UDFs to produce millions of rows across domains like e-commerce and IoT, writing outputs directly to Parquet, JSON, CSV, or Delta formats.

What is the best way to create synthetic datasets with referential integrity in Unity Catalog?▼

Creating synthetic datasets with referential integrity in Unity Catalog involves providing your catalog and schema, letting the generation process create missing schemas or volumes, and enforcing relational coherence across story-driven domain tables.

Can I use Faker with Spark Pandas UDFs for synthetic data generation on serverless compute?▼

Faker works with Spark Pandas UDFs on serverless compute to parallelize synthetic data generation across thousands to millions of rows, intentionally avoiding cache and persist operations to maintain serverless compatibility.

How do I output synthetic e-commerce and IoT datasets to Delta format in Databricks?▼

Output synthetic e-commerce and IoT datasets to Delta format in Databricks by configuring Spark and Faker generation jobs to write story-driven, realistically distributed records directly into specified Unity Catalog volumes.

Does synthetic data generation on Databricks serverless compute have limitations with caching?▼

Synthetic data generation on Databricks serverless compute has limitations requiring the avoidance of cache and persist operations, with fallback options available for classic clusters or local generation for very small datasets.

Related Skills

Explore More Data & Analytics Skills→
ThomazRossitoThomazRossito
31

databricks-synthetic-data-gen

Generate story-driven synthetic datasets in Databricks with Spark and Faker.

Community
Advanced
jluo41jluo41

haipipe-data-1-source

Convert CSV, XML, Parquet, and JSON files into standardized SourceSet assets.

Community
Advanced
Zack2626-okZack2626-ok

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Spark Declarative Pipelines.

Community
Advanced
Zack2626-okZack2626-ok

databricks-synthetic-data-gen

Generate realistic synthetic data for Databricks using Spark and Faker.

Community
Advanced