databricks-synthetic-data-gen

Generate synthetic data with Spark and Faker into Unity Catalog volumes.

11|3|Updated Jun 10, 2025

Search Tags:#faker #spark #databricks #unity-catalog #synthetic-data #pandas-udf

Ready to run

npx skills add https://github.com/Paldom/databricks-apps-fastapi-starter --skill databricks-synthetic-data-gen-paldom

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/Paldom/databricks-apps-fastapi-starter/tree/main/.gemini/skills/databricks-synthetic-data-gen
Command: npx skills add https://github.com/Paldom/databricks-apps-fastapi-starter --skill databricks-synthetic-data-gen-paldom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyspark, numpy, pandas, faker, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Generate realistic synthetic data for Databricks environments using Spark + Faker to enable scalable testing, demos, and development without touching production data.

Core Features & Use Cases

Serverless execution with Spark + Faker + Pandas UDFs for scalable generation.
Output formats include Parquet, JSON, CSV, and Delta tables, with volume-backed storage via Unity Catalog.
Optional local generation for very small datasets and quick prototyping, with seamless upload to volumes.

Quick Start

Run the generate_synthetic_data.py script to generate synthetic data and store it in the configured Unity Catalog volume.

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate synthetic data in Databricks using Spark and Faker?▼

You can generate synthetic data in Databricks by running a Python script that uses Spark and Faker with Pandas UDFs to create realistic datasets, storing the output in Unity Catalog volumes.

What output formats are supported when generating synthetic data for Unity Catalog?▼

Supported output formats for synthetic data generation include Parquet, JSON, CSV, and Delta tables, using volume-backed storage managed automatically through Unity Catalog.

Can I use Pandas UDFs with Faker for scalable synthetic data generation on serverless compute?▼

Yes, scalable synthetic data generation uses Pandas UDFs alongside the Faker library to run efficiently on Databricks serverless compute, enabling large dataset creation for testing.

Does synthetic data generation require an existing schema or volume in Unity Catalog?▼

No, the synthetic data generation script creates the necessary schema and Unity Catalog volumes in-script, so you do not need to manually provision storage before running it.

What is the best way to create realistic patterns like holidays in synthetic data?▼

To create realistic patterns in synthetic data, the generation script supports optional holiday inputs alongside Faker, allowing the output to accurately reflect real-world temporal variations.

Can I generate small datasets locally before uploading to Databricks volumes?▼

Yes, the tool supports optional local generation for very small datasets and quick prototyping, with seamless subsequent upload of the generated files to Unity Catalog volumes.