databricks-synthetic-data-generation

Generate synthetic customer, order, and ticket datasets with Faker and Spark for Databricks volumes.

1|1|Updated Oct 1, 2025

Search Tags:#generate #faker #spark #data-generation #databricks #synthetic-data

Ready to run

npx skills add https://github.com/mkgs-databricks-demos/synthea-on-fhir --skill databricks-synthetic-data-generation-mkgs-databricks-demos

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-synthetic-data-generation
Source: https://github.com/mkgs-databricks-demos/synthea-on-fhir/tree/main/.cursor/skills/databricks-synthetic-data-generation
Command: npx skills add https://github.com/mkgs-databricks-demos/synthea-on-fhir --skill databricks-synthetic-data-generation-mkgs-databricks-demos

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Generating realistic, story-driven synthetic data for analytics environments typically requires complex setup and careful tuning; this skill automates the process using Faker and Spark to produce coherent data that can power test datasets and demos in Databricks.

Core Features & Use Cases

Realistic data generation: Create customer, order, and ticket datasets with non-linear distributions and reproducible seeds.
Databricks integration: Write results to a Databricks Volume as Parquet files for downstream SDP pipelines.
Deterministic workflows: Reuse Spark contexts and manage infrastructure creation within a script to streamline debugging and iteration.

Quick Start

Run scripts/generate_data.py to generate the datasets in your Databricks environment after configuring N_CUSTOMERS, N_ORDERS, and N_TICKETS.

Frequently Asked Questions about databricks-synthetic-data-generation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate realistic synthetic data for testing Databricks pipelines?▼

You can generate realistic synthetic data for Databricks pipelines using Faker and Spark to simulate non-linear patterns. This approach creates coherent test datasets with reproducible seeds, writing the output as Parquet files directly to Databricks volumes for downstream processing.

What is the best way to create non-linear distributions in synthetic Spark datasets?▼

Creating non-linear distributions in synthetic Spark datasets is achieved by combining Faker with Spark context generation. This method produces realistic customer, order, and ticket data patterns, ensuring your test data accurately reflects real-world analytics scenarios.

Can I write synthetic data generation outputs directly to a Databricks Volume?▼

Yes, you can write synthetic data generation outputs directly to a Databricks Volume. The reproducible script-based workflow formats the generated records as Parquet files, storing them in configured volumes for seamless integration with downstream data pipelines.

Do I need to configure specific parameters to run a synthetic data generation script in Spark?▼

You need to configure parameters like N_CUSTOMERS, N_ORDERS, and N_TICKETS to run a synthetic data generation script in Spark. Setting these variables defines the scale of your dataset, allowing the deterministic workflow to produce the exact volume of test records required.

How does a reproducible seed improve synthetic data generation for analytics dashboards?▼

A reproducible seed improves synthetic data generation by ensuring deterministic outputs across multiple runs. Reusing Spark contexts with managed infrastructure allows you to generate identical customer and order datasets, streamlining debugging and validation for analytics dashboards.

When should I use Faker instead of random Spark functions for test data generation?▼

You should use Faker instead of random Spark functions when you need coherent, story-driven test data rather than arbitrary values. Faker generates realistic entities like customer profiles and tickets with non-linear distributions, providing higher quality validation datasets for analytics environments.