databricks-synthetic-data-gen

Generate realistic synthetic data for Databricks using Spark, Faker, and Pandas UDFs.

Updated Mar 23, 2024

Search Tags:#data generation #Spark #synthetic data #Faker #Databricks

Ready to run

npx skills add https://github.com/m19c/dotfiles --skill databricks-synthetic-data-gen-m19c

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/m19c/dotfiles/tree/main/claude/.claude/skills/databricks-synthetic-data-gen
Command: npx skills add https://github.com/m19c/dotfiles --skill databricks-synthetic-data-gen-m19c

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-connect, faker, numpy, pandas, holidays, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the generation of realistic synthetic data for Databricks, enabling you to quickly populate Databricks environments with large volumes of realistic data for testing, training, and development purposes.

Core Features & Use Cases

Synthetic Data Generation: Supports serverless execution, multiple output formats (Parquet/JSON/CSV/Delta), and scales from thousands to millions of rows.
Data Patterns: Generates realistic data patterns, including referential integrity, time patterns, and domain-specific patterns.
Output Formats: Outputs data in Parquet, JSON, CSV, or Delta formats for easy integration with various tools and platforms.
Use Case: Generate synthetic customer data for a retail e-commerce scenario or synthetic sensor data for IoT applications.

Quick Start

Run the following command to generate synthetic data for a specified catalog and schema:

databricks-synthetic-data-gen generate_data.py --catalog <your_catalog> --schema <your_schema>

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate realistic synthetic data in Databricks for testing?▼

You generate realistic synthetic data in Databricks by running a Python script that leverages Spark, Faker, and Pandas UDFs to populate environments with realistic patterns. It supports serverless execution and multiple output formats like Delta and Parquet.

Can I generate millions of rows of synthetic data using Spark and Faker?▼

Yes, you can generate millions of rows of synthetic data using Spark and Faker. The process scales seamlessly from thousands to millions of rows through distributed serverless execution, utilizing Pandas UDFs for efficient data generation across your cluster.

What output formats are supported when creating synthetic datasets for Databricks?▼

Supported output formats for synthetic datasets include Parquet, JSON, CSV, and Delta. This allows easy integration with various tools and platforms within the Databricks ecosystem for your testing and development needs.

Do I need databricks-connect installed to generate synthetic data locally?▼

Yes, databricks-connect is required to generate synthetic data and interact with your Databricks environment. Additionally, you need the faker, numpy, pandas, and holidays Python packages installed to execute the generation scripts successfully.

Can I generate small synthetic datasets locally and upload them to Databricks volumes?▼

Yes, for small datasets you can optionally generate the synthetic data locally and then upload it directly to Databricks volumes. This provides a faster alternative for smaller testing sets before scaling up to serverless Spark execution.