databricks-synthetic-data-generation

Write a Python script using Faker and pandas to build DataFrames, then execute it on a Databricks cluster with the run_python_file_on_databricks MCP tool. Convert the pandas DataFrames to Spark and save them as parquet files in a Unity Catalog volume.

Faker generates realistic names, companies, and dates, while the holidays library provides country-specific calendars for realistic date patterns. numpy supplies log-normal, Pareto, and exponential distributions, and pandas handles row-by-row generation before Spark saves the output.

Save generated data as parquet files in a Unity Catalog volume rather than tables. This raw data becomes the input for a downstream Spark Declarative Pipeline that builds bronze, silver, and gold layers with aggregations.

Uniform distributions and insufficient row counts cause patterns to disappear after GROUP BY operations. Use log-normal or exponential distributions, weighted categories, and at least 10K-50K rows so trends survive daily, weekly, and per-category aggregation.

Edit the local Python file to fix the error, then re-execute with the same cluster_id and context_id returned by the first run. Reusing the context keeps installed libraries like Faker available and runs much faster than starting fresh.