synthetic-data-generation

Generate realistic synthetic data for Databricks workflows using PySpark and Faker.

Updated Sep 29, 2025

Search Tags:#faker #spark #parquet #data-generation #databricks #synthetic-data #datapipelines

Ready to run

npx skills add https://github.com/juanlamadrid20/dbrx-multi-agent-retail-intelligence --skill synthetic-data-generation-juanlamadrid20

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: synthetic-data-generation
Source: https://github.com/juanlamadrid20/dbrx-multi-agent-retail-intelligence/tree/main/.claude/skills/synthetic-data-generation
Command: npx skills add https://github.com/juanlamadrid20/dbrx-multi-agent-retail-intelligence --skill synthetic-data-generation-juanlamadrid20

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, pandas, faker, holidays, pyspark.

What problem does it solve?

Generate realistic synthetic data for Databricks workflows to test and demo data pipelines while preserving privacy and control over data quality.

Core Features & Use Cases

Faker-based data generation with non-linear distributions, deterministic seeding, and referential integrity across tables.
Spark-ready outputs saved as Parquet (or Delta) in a Databricks volume for downstream SDP pipelines.
Configurable date ranges, volume patterns, and scenario-driven events (holidays, weekends, spikes) to support demos, testing, and privacy-safe datasets.

Quick Start

Create a Python script using Faker and Spark (e.g., scripts/generate_data.py), run it on Databricks, and save the results as Parquet files.

Frequently Asked Questions about synthetic-data-generation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate realistic synthetic data for testing Spark pipelines in Databricks?▼

Generate realistic synthetic data for Spark pipelines by using Faker with PySpark to create privacy-safe test datasets. The Skill supports non-linear distributions, deterministic seeding, and referential integrity, saving outputs as Parquet files directly to a Databricks volume.

Can I simulate holiday spikes and weekend volume patterns when creating synthetic datasets?▼

Yes, you can simulate holiday spikes and weekend volume patterns when creating synthetic datasets. The Skill uses the holidays library alongside configurable date ranges and scenario-driven events to generate realistic non-linear data distributions for testing.

Does this synthetic data generation approach support deterministic seeding for reproducible test runs?▼

Yes, this synthetic data generation approach supports deterministic seeding to ensure reproducible test runs. By setting a fixed seed within the Faker and PySpark data generation logic, you can consistently reproduce identical synthetic datasets across multiple pipeline executions.

Do I need PySpark and Faker installed to create Parquet files for Databricks volumes?▼

Yes, you need PySpark and Faker installed to create Parquet files for Databricks volumes. The Skill also requires numpy, pandas, and holidays libraries to generate realistic synthetic data with proper distributions and scenario-driven events.

What is the best way to maintain referential integrity across multiple synthetic tables in Databricks?▼

The best way to maintain referential integrity across multiple synthetic tables in Databricks is to use a generation script that enforces relational mappings. This Skill creates linked Spark-ready outputs saved as Parquet or Delta, ensuring downstream SDP pipelines process consistent data.

Why use Faker with PySpark instead of standard random sampling for data pipeline testing?▼

Using Faker with PySpark provides realistic, locale-aware data like names and addresses that standard random sampling cannot generate. It ensures privacy-safe datasets while applying non-linear distributions and holiday events, producing higher quality test data for Databricks workflows.