skills.rest
RDEWAI
RDEWAIOFFICIAL
Explore all skills by this author →
GitHub

create-ingestion

Generate config-driven Bronze ingestion code for Spark and Delta Lake pipelines.

5|1|Updated Sep 23, 2025
Search Tags:#config-driven#spark#data-quality#data-engineering#delta-lake#unity-catalog#bronze-ingestion
One-click install
npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-ingestion
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: create-ingestion
Source: https://github.com/RDEWAI/Redefining-DataEngineering-With-AI/tree/main/chapter-6/developer-plugin/skills/create-ingestion
Command: npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-ingestion

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the tedious, error-prone manual work of writing Bronze ingestion code for data engineering pipelines, ensuring all generated code strictly adheres to project low-level design specifications, coding standards, and data quality requirements without repetitive boilerplate work.

Core Features & Use Cases

  • Config-Driven Code Generation: Produces production-ready Bronze ingestion runner, factory, SparkSubmit wrapper, and per-table YAML configs from approved LLD, DMS, STM, DQS, and user stories, with no hardcoded project identifiers.
  • Dual Mode Operation: Supports story mode (generate only deliverables for a specific user story) and full mode (generate the entire ingestion layer for the project), with automatic routing of out-of-scope deliverables to other specialized skills.
  • Use Case: A data engineer implementing a new data pipeline story can use this Skill to automatically generate all required Bronze ingestion code, validate upstream dependency completion, and ensure compliance with project standards in minutes instead of hours.

Quick Start

Use the create-ingestion skill to generate Bronze ingestion code for your data pipeline by providing a valid user story ID or enabling full project generation mode.

Frequently Asked Questions about create-ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate Bronze ingestion code generation for Spark and Delta Lake pipelines?▼

Automating Bronze ingestion code generation for Spark and Delta Lake pipelines involves using config-driven templates that produce production-ready runners, factories, and per-table YAML configs from approved source-to-target mappings and user stories.

What is config-driven data ingestion in Delta Lake and Unity Catalog?▼

Config-driven data ingestion in Delta Lake and Unity Catalog is a pattern where per-table YAML files define source-to-target mappings, allowing automated generation of idempotent partitioned writes and Spark Expectations data quality checks without hardcoded project identifiers.

How do I generate Spark Bronze layer pipelines from low-level design documents?▼

Generating Spark Bronze layer pipelines from low-level design documents requires routing approved LLD specifications into an automated generator that validates upstream dependency completion, enforces coding standards, and outputs SparkSubmit wrappers aligned with the design.

Does Spark Expectations work with config-driven Delta Lake ingestion for data quality?▼

Spark Expectations works with config-driven Delta Lake ingestion by integrating validation rules directly into the generated pipeline code, enforcing data quality checks during idempotent partitioned writes to the Bronze layer.

Can I generate the entire data engineering ingestion layer for a project at once?▼

Generating the entire data engineering ingestion layer for a project at once is supported through full mode operation, which produces all required Bronze ingestion runners and per-table configs while automatically routing out-of-scope deliverables to other specialized skills.

Why validate upstream dependency completion before generating Bronze ingestion code?▼

Validating upstream dependency completion before generating Bronze ingestion code ensures that required source tables, schema mappings, and low-level design documents are fully approved, preventing runtime pipeline failures and ensuring strict compliance with project standards.

Related Skills

Explore More Data & Analytics Skills→
RDEWAIRDEWAI
5

approve-hld

Approve high-level design documents by updating status and version history.

Official
Intermediate
manutejmanutej
61

apache-spark-data-processing

Teach Apache Spark data processing across RDDs, DataFrames, Spark SQL, streaming, and MLlib.

Community
Advanced
RajuNaikRajuNaik
1

databricks-medallion-architecture

Design and implement Bronze/Silver/Gold medallion data architectures on Databricks.

Community
Advanced
RDEWAIRDEWAI
5

update-ingestion

Update Bronze ingestion artifacts to match LLD and STM specifications.

Official
Advanced