skills.rest
ThomazRossito
ThomazRossitoCOMMUNITY
Explore all skills by this author →
Website𝕏@thomazrossitoGitHub

databricks-synthetic-data-gen

Generate story-driven synthetic datasets in Databricks with Spark and Faker.

31|18|Updated Apr 4, 2026
Search Tags:#delta#faker#spark#databricks#unity-catalog#synthetic-data#pandas-udf
One-click install
npx skills add https://github.com/ThomazRossito/data-agents --skill databricks-synthetic-data-gen-thomazrossito
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/ThomazRossito/data-agents/tree/main/skills/databricks/databricks-synthetic-data-gen
Command: npx skills add https://github.com/ThomazRossito/data-agents --skill databricks-synthetic-data-gen-thomazrossito

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-connect, faker, numpy, pandas, holidays, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Generate realistic, story-driven synthetic datasets that demonstrate business impact, preserve referential integrity, and scale on Databricks so teams can demo analytics, validate pipelines, train models, and test data quality without exposing production data. The Skill enforces domain coherence, non-uniform distributions, event-driven anomalies (incidents/outages), and explicit user confirmation of Unity Catalog targets to avoid accidental writes.

Core Features & Use Cases

  • Scalable generation using Spark + Faker + pandas_udf patterns that scale from thousands to millions of rows with partitioning guidance.
  • Story-first datasets: designs data to tell a business story (incident → impact → analysis → fix) with skewed distributions, time patterns, and correlated attributes.
  • Databricks-native output: supports Parquet/JSON/CSV to Volumes and managed Delta tables in Unity Catalog; always asks for catalog and schema and creates schema/volume if missing.
  • Serverless-first execution with databricks-connect patterns and explicit rules (no .cache()/.persist(), write master tables then read for FK joins).
  • Validation & troubleshooting guidance: pre-generation planning, post-generation checks, and troubleshooting for common environment issues and UDF dependency management.

Quick Start

Generate a 100000-row synthetic support-demo dataset in catalog my_catalog.my_schema using Spark + Faker, write master tables to Delta and raw data to /Volumes/my_catalog/my_schema/raw_data, and present the plan and table specs for my approval.

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate realistic synthetic data in Databricks using Spark and Faker?▼

To generate synthetic data in Databricks, this Skill uses Spark combined with Faker via pandas UDFs to create story-driven datasets with referential integrity, writing outputs as Delta tables or Parquet and JSON files into Unity Catalog volumes.

Can I create synthetic datasets on Databricks serverless compute without caching issues?▼

Yes, synthetic data generation runs on serverless compute by avoiding .cache() and .persist() calls, writing master tables to Delta first, then reading them back to perform foreign key joins for referential integrity.

How do I scale synthetic data generation to millions of rows in Unity Catalog?▼

Synthetic data generation scales to millions of rows in Unity Catalog by applying Spark partitioning guidance alongside pandas UDF Faker implementations, ensuring distributed processing across your Databricks cluster.

What is story-driven synthetic data and how does it help with analytics demos?▼

Story-driven synthetic data designs datasets to tell a business narrative like incident to impact to analysis with skewed distributions, time patterns, and correlated attributes, enabling realistic analytics demos without exposing production data.

Do I need to manually create schemas and volumes before generating synthetic data in Unity Catalog?▼

No, you do not need to manually create schemas and volumes beforehand; the synthetic data generation process requires a user-provided catalog and schema, then automatically creates the schema and volume in-script if they are missing.

Why should I use synthetic data instead of production data for ML training and pipeline testing?▼

You should use synthetic data for ML training and pipeline testing to avoid exposing sensitive production data while maintaining domain coherence, non-uniform distributions, and event-driven anomalies that validate data quality and model performance.

Related Skills

Explore More Data & Analytics Skills→
leary-pokenleary-poken
1

databricks-synthetic-data-gen

Generate scalable synthetic data for Databricks workloads using Spark and Faker.

Community
Advanced
LaurentPRAT-DBLaurentPRAT-DB

synthetic-data-generation

Generate realistic synthetic data with Faker and Spark for Databricks Volumes.

Community
Advanced
PaldomPaldom
11

databricks-synthetic-data-gen

Generate synthetic data with Spark and Faker into Unity Catalog volumes.

Community
Advanced
alessandro9110alessandro9110
4

databricks-synthetic-data-generation

Generate synthetic transactional data with Faker and Spark for Databricks.

Community
Advanced