skills.rest
m19c
m19cCOMMUNITY
Explore all skills by this author →
𝕏@m19c_GitHub

databricks-synthetic-data-gen

Generate realistic synthetic data for Databricks using Spark, Faker, and Pandas UDFs.

Updated Mar 23, 2024
Search Tags:#data generation#Spark#synthetic data#Faker#Databricks
One-click install
npx skills add https://github.com/m19c/dotfiles --skill databricks-synthetic-data-gen-m19c
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-synthetic-data-gen
Source: https://github.com/m19c/dotfiles/tree/main/claude/.claude/skills/databricks-synthetic-data-gen
Command: npx skills add https://github.com/m19c/dotfiles --skill databricks-synthetic-data-gen-m19c

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-connect, faker, numpy, pandas, holidays, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the generation of realistic synthetic data for Databricks, enabling you to quickly populate Databricks environments with large volumes of realistic data for testing, training, and development purposes.

Core Features & Use Cases

  • Synthetic Data Generation: Supports serverless execution, multiple output formats (Parquet/JSON/CSV/Delta), and scales from thousands to millions of rows.
  • Data Patterns: Generates realistic data patterns, including referential integrity, time patterns, and domain-specific patterns.
  • Output Formats: Outputs data in Parquet, JSON, CSV, or Delta formats for easy integration with various tools and platforms.
  • Use Case: Generate synthetic customer data for a retail e-commerce scenario or synthetic sensor data for IoT applications.

Quick Start

Run the following command to generate synthetic data for a specified catalog and schema:

databricks-synthetic-data-gen generate_data.py --catalog <your_catalog> --schema <your_schema>

Frequently Asked Questions about databricks-synthetic-data-gen

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate realistic synthetic data in Databricks for testing?▼

You generate realistic synthetic data in Databricks by running a Python script that leverages Spark, Faker, and Pandas UDFs to populate environments with realistic patterns. It supports serverless execution and multiple output formats like Delta and Parquet.

Can I generate millions of rows of synthetic data using Spark and Faker?▼

Yes, you can generate millions of rows of synthetic data using Spark and Faker. The process scales seamlessly from thousands to millions of rows through distributed serverless execution, utilizing Pandas UDFs for efficient data generation across your cluster.

What output formats are supported when creating synthetic datasets for Databricks?▼

Supported output formats for synthetic datasets include Parquet, JSON, CSV, and Delta. This allows easy integration with various tools and platforms within the Databricks ecosystem for your testing and development needs.

Do I need databricks-connect installed to generate synthetic data locally?▼

Yes, databricks-connect is required to generate synthetic data and interact with your Databricks environment. Additionally, you need the faker, numpy, pandas, and holidays Python packages installed to execute the generation scripts successfully.

Can I generate small synthetic datasets locally and upload them to Databricks volumes?▼

Yes, for small datasets you can optionally generate the synthetic data locally and then upload it directly to Databricks volumes. This provides a faster alternative for smaller testing sets before scaling up to serverless Spark execution.

Related Skills

Explore More Data & Analytics Skills→
datasciencemonkeydatasciencemonkey
27

synthetic-data-generation

Generate realistic synthetic datasets for Databricks with Faker, numpy, and Spark.

Community
Advanced
juanlamadrid20juanlamadrid20

databricks-synthetic-data-generation

Generate realistic synthetic data for Databricks using Python, Faker, and Spark.

Community
Advanced
leary-pokenleary-poken
1

databricks-synthetic-data-gen

Generate scalable synthetic data for Databricks workloads using Spark and Faker.

Community
Advanced
JingyiWngJingyiWng

databricks-synthetic-data-gen

Generate realistic synthetic datasets with Spark and Faker for Databricks.

Community
Advanced