synthetic-data-generation

Generate realistic synthetic data for Spark pipelines by using Faker with PySpark to create privacy-safe test datasets. The Skill supports non-linear distributions, deterministic seeding, and referential integrity, saving outputs as Parquet files directly to a Databricks volume.

Yes, you can simulate holiday spikes and weekend volume patterns when creating synthetic datasets. The Skill uses the holidays library alongside configurable date ranges and scenario-driven events to generate realistic non-linear data distributions for testing.

Yes, this synthetic data generation approach supports deterministic seeding to ensure reproducible test runs. By setting a fixed seed within the Faker and PySpark data generation logic, you can consistently reproduce identical synthetic datasets across multiple pipeline executions.

Yes, you need PySpark and Faker installed to create Parquet files for Databricks volumes. The Skill also requires numpy, pandas, and holidays libraries to generate realistic synthetic data with proper distributions and scenario-driven events.

The best way to maintain referential integrity across multiple synthetic tables in Databricks is to use a generation script that enforces relational mappings. This Skill creates linked Spark-ready outputs saved as Parquet or Delta, ensuring downstream SDP pipelines process consistent data.

Using Faker with PySpark provides realistic, locale-aware data like names and addresses that standard random sampling cannot generate. It ensures privacy-safe datasets while applying non-linear distributions and holiday events, producing higher quality test data for Databricks workflows.