databricks-synthetic-data-generation

Generate configurable Faker-based synthetic datasets as Parquet in Databricks volumes.

Updated Sep 9, 2017

Search Tags:#testing #faker #spark #parquet #databricks #synthetic-data

Ready to run

npx skills add https://github.com/mirakui/dotfiles --skill databricks-synthetic-data-generation-mirakui

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-synthetic-data-generation
Source: https://github.com/mirakui/dotfiles/tree/main/claude/skills/databricks-synthetic-data-generation
Command: npx skills add https://github.com/mirakui/dotfiles --skill databricks-synthetic-data-generation-mirakui

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Generate realistic synthetic data for Databricks test environments, enabling safe, repeatable testing without exposing real data.

Core Features & Use Cases

Faker-based data generation for customers, orders, tickets with configurable volumes.
Spark/Databricks-ready outputs (Parquet) stored in a catalog/schema volume to feed SDP pipelines.
Reproducible runs with deterministic seeds and date ranges for stable demos and QA.

Quick Start

Create a local Python script at scripts/generate_data.py with your configuration and run it on Databricks using the provided tooling.

Frequently Asked Questions about databricks-synthetic-data-generation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate synthetic data for Databricks test environments?▼

To generate synthetic data for Databricks, you can use this Skill to create configurable datasets with varying sizes and distributions using Faker-based generation, then persist them as Parquet files in a catalog/schema volume for QA testing.

Can I create reproducible synthetic datasets in Spark for stable demos?▼

Yes, you can create reproducible synthetic datasets in Spark by configuring deterministic seeds and specific date ranges, which ensures stable and repeatable outputs for demos and QA pipelines.

Does this approach support generating raw, non-aggregated data for SDP pipelines?▼

Yes, this approach supports generating raw, non-aggregated data for SDP pipelines by using Faker-based logic to create customer, order, and ticket records, storing them directly as Parquet volumes in your Databricks catalog.

What is the best way to create configurable Parquet volumes for Databricks QA testing?▼

The best way to create configurable Parquet volumes for Databricks QA testing is using a local Python script with Faker-based data generation, executed on Databricks to persist records with varying sizes and distributions into a configurable schema.

Do I need to configure a specific catalog and schema to store synthetic data in Databricks?▼

Yes, you need to configure a specific catalog and schema because this Skill uses Spark-based persistence to write the generated synthetic data directly into Parquet volumes within your defined Databricks catalog and schema structure.