databricks-synthetic-data-gen

Generate scalable synthetic data in Databricks using Spark combined with Faker and Pandas UDFs. This approach scales from thousands to millions of rows while enforcing non-linear distributions, time-based patterns, and referential integrity across domains like e-commerce, CRM, or IoT.

Yes, synthetic data can be output directly to Delta, Parquet, JSON, or CSV formats. Small datasets generate locally for upload to Unity Catalog volumes, while large datasets generate directly into volumes using serverless compute by default.

Maintain referential integrity in synthetic datasets by enforcing master-detail generation patterns. This ensures realistic relationships between entities, such as customers and orders, while applying non-linear distributions and time-based patterns for accurate domain modeling.

Yes, the synthetic data generation process uses serverless compute by default for Databricks workloads. This allows efficient scaling from small local uploads to massive datasets generated directly within Unity Catalog volumes.

Create mock datasets for Databricks demos and sandbox testing by running the generator to produce representative synthetic data. A quick start example generates 10,000 customers and 50,000 orders directly into your Unity Catalog volume without exposing real data.

Scalable synthetic data generation with Spark requires the faker, pandas, numpy, and holidays Python dependencies. These libraries support Pandas UDFs to generate realistic, non-linear distributions and time-based patterns across large datasets.