skills.rest
Paldom
PaldomCOMMUNITY
Explore all skills by this author →
GitHub

databricks-synthetic-data-gen

Generate synthetic data with Spark and Faker into Unity Catalog volumes.

11|3|Updated Jun 10, 2025
Search Tags:#faker#spark#databricks#unity-catalog#synthetic-data#pandas-udf
One-click install
npx skills add https://github.com/Paldom/databricks-apps-fastapi-starter --skill databricks-synthetic-data-gen-paldom
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/Paldom/databricks-apps-fastapi-starter/tree/main/.gemini/skills/databricks-synthetic-data-gen
Command: npx skills add https://github.com/Paldom/databricks-apps-fastapi-starter --skill databricks-synthetic-data-gen-paldom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyspark, numpy, pandas, faker, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Generate realistic synthetic data for Databricks environments using Spark + Faker to enable scalable testing, demos, and development without touching production data.

Core Features & Use Cases

  • Serverless execution with Spark + Faker + Pandas UDFs for scalable generation.
  • Output formats include Parquet, JSON, CSV, and Delta tables, with volume-backed storage via Unity Catalog.
  • Optional local generation for very small datasets and quick prototyping, with seamless upload to volumes.

Quick Start

Run the generate_synthetic_data.py script to generate synthetic data and store it in the configured Unity Catalog volume.

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate synthetic data in Databricks using Spark and Faker?▼

You can generate synthetic data in Databricks by running a Python script that uses Spark and Faker with Pandas UDFs to create realistic datasets, storing the output in Unity Catalog volumes.

What output formats are supported when generating synthetic data for Unity Catalog?▼

Supported output formats for synthetic data generation include Parquet, JSON, CSV, and Delta tables, using volume-backed storage managed automatically through Unity Catalog.

Can I use Pandas UDFs with Faker for scalable synthetic data generation on serverless compute?▼

Yes, scalable synthetic data generation uses Pandas UDFs alongside the Faker library to run efficiently on Databricks serverless compute, enabling large dataset creation for testing.

Does synthetic data generation require an existing schema or volume in Unity Catalog?▼

No, the synthetic data generation script creates the necessary schema and Unity Catalog volumes in-script, so you do not need to manually provision storage before running it.

What is the best way to create realistic patterns like holidays in synthetic data?▼

To create realistic patterns in synthetic data, the generation script supports optional holiday inputs alongside Faker, allowing the output to accurately reflect real-world temporal variations.

Can I generate small datasets locally before uploading to Databricks volumes?▼

Yes, the tool supports optional local generation for very small datasets and quick prototyping, with seamless subsequent upload of the generated files to Unity Catalog volumes.

Related Skills

Explore More Data & Analytics Skills→
leary-pokenleary-poken
1

databricks-synthetic-data-gen

Generate scalable synthetic data for Databricks workloads using Spark and Faker.

Community
Advanced
LaurentPRAT-DBLaurentPRAT-DB

synthetic-data-generation

Generate realistic synthetic data with Faker and Spark for Databricks Volumes.

Community
Advanced
teegin-gteegin-g

databricks-synthetic-data-gen

Generate scalable synthetic data for Databricks using Spark and Faker.

Community
Advanced
ThomazRossitoThomazRossito
31

databricks-synthetic-data-gen

Generate story-driven synthetic datasets in Databricks with Spark and Faker.

Community
Advanced