databricks-synthetic-data-gen

Name: databricks-synthetic-data-gen
Availability: InStock
Author: aaronachermann

Generate story-driven synthetic datasets for Databricks with Spark and Faker UDFs.

Updated Apr 18, 2026

Search Tags:#testing #faker #spark #data-generation #databricks #unity-catalog #synthetic-data

Ready to run

npx skills add https://github.com/aaronachermann/PolentaEncoders --skill databricks-synthetic-data-gen-aaronachermann

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/aaronachermann/PolentaEncoders/tree/main/.github/skills/databricks-synthetic-data-gen
Command: npx skills add https://github.com/aaronachermann/PolentaEncoders --skill databricks-synthetic-data-gen-aaronachermann

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires faker, pandas, numpy, databricks-connect, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps teams quickly create realistic synthetic datasets for Databricks to test pipelines, run demos, and validate analytics without exposing real customer data.

Core Features & Use Cases

Spark + Faker data generation for realistic, story-driven datasets
Serverless-ready and scalable to millions of rows
Multiple output formats (Parquet/Delta/JSON/CSV) and Unity Catalog friendly schemas
Data storytelling patterns with referential integrity and measurable business impact

Quick Start

Configure the data story you want (for example a 100000-row customers and orders dataset with skewed tiers and a simulated outage) and run the generator to produce raw data in your catalog volume.

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate realistic synthetic data in Databricks for testing pipelines?▼

Generate realistic synthetic data in Databricks using Spark and Faker UDFs to create story-driven datasets with non-uniform distributions and referential integrity. This allows you to test pipelines and run demos without exposing real customer data.

Can I create synthetic datasets with referential integrity for Databricks dashboards?▼

Yes, you can create synthetic datasets with referential integrity across customers, orders, and incidents. The generator enforces relationships and measurable business impacts, making the output suitable for dashboards and ML models.

Does synthetic data generation for Databricks support Unity Catalog integration?▼

Yes, synthetic data generation for Databricks supports Unity Catalog integration with planning prompts and friendly schemas. It outputs raw data directly to your catalog volume in Delta, Parquet, JSON, or CSV formats.

What is the best way to scale synthetic data generation to millions of rows on Databricks?▼

The best way to scale synthetic data generation to millions of rows on Databricks is using serverless-ready Spark and Faker UDFs. This approach efficiently produces large datasets while maintaining skewed distributions and story-driven patterns.

Do I need Databricks-connect to run the synthetic data generator?▼

Yes, you need databricks-connect along with faker, pandas, and numpy dependencies to run the synthetic data generator. These libraries enable serverless execution and the creation of story-driven datasets for root-cause analysis.