skills.rest
RDEWAI
RDEWAIOFFICIAL
Explore all skills by this author →
GitHub

update-ingestion

Update Bronze ingestion artifacts to match LLD and STM specifications.

5|1|Updated Sep 23, 2025
Search Tags:#data quality#data engineering#delta lake#pipeline maintenance#bronze ingestion#lld sync#spark expectations
One-click install
npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill update-ingestion
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: update-ingestion
Source: https://github.com/RDEWAI/Redefining-DataEngineering-With-AI/tree/main/chapter-6/developer-plugin/skills/update-ingestion
Command: npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill update-ingestion

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the risk of manual, error-prone full rewrites of Bronze ingestion artifacts when Low-Level Design (LLD) or Story Technical Model (STM) specifications change, ensuring data pipelines stay aligned with design updates without disrupting active runs.

Core Features & Use Cases

  • 8 Targeted Update Scenarios: Handles adding/removing/renaming Bronze tables, adjusting per-table ingestion knobs (retries, timeouts, empty-input behavior), updating runner/factory logic, refreshing data quality rules, updating Spark expectations runners, and generating plain SQL Bronze DDL migrations.
  • Minimal Incremental Edits: Only modifies the specific files impacted by design changes, preserving all unrelated ingestion code and configuration to avoid breaking existing pipeline functionality.
  • Built-in Validation: Automatically enforces project-specific coding patterns, library version requirements, and test coverage, with mandatory linting and test runs before completion.
  • Real-World Use Case: For example, if the LLD adds a new patient_vitals Bronze table, the skill automatically creates the corresponding YAML config, updates the ingestion factory if needed, adds required test coverage, and generates the matching Delta Lake DDL migration without touching unrelated ingestion components.

Quick Start

Use the update-ingestion skill to add the new patient_vitals Bronze table defined in the latest LLD.

Frequently Asked Questions about update-ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a new Bronze table to a Delta Lake ingestion pipeline without breaking existing runs?▼

To add a new Bronze table safely, sync the updated Low-Level Design to generate the specific YAML config, update the ingestion factory, add test coverage, and generate the matching DDL migration while preserving unrelated ingestion components.

What is the safest way to update per-table ingestion parameters like retries and timeouts?▼

Updating per-table ingestion parameters safely requires applying minimal incremental edits to the specific table's YAML configuration, ensuring the active data pipeline remains uninterrupted and unrelated ingestion code stays untouched.

How do I sync data quality rules and Spark expectations with new Story Technical Model specifications?▼

Syncing data quality rules with Story Technical Model updates involves refreshing the specific rule sets and updating the Spark expectations runner, followed by mandatory linting and test runs to enforce project-specific coding patterns.

Can I generate plain SQL DDL migrations for the Bronze layer from my evolving LLD?▼

Yes, you can generate plain SQL Bronze layer DDL migrations directly from evolving LLD specifications. This applies incremental updates to the Delta Lake schema without requiring a full rewrite of the ingestion artifacts.

Why should I use incremental edits instead of a full rewrite for Bronze ingestion pipeline maintenance?▼

Incremental edits prevent the risk of manual, error-prone full rewrites by modifying only the specific files impacted by LLD or STM design changes, preserving existing pipeline functionality and enforcing pinned library versions.

Related Skills

Explore More Data & Analytics Skills→
RDEWAIRDEWAI
5

validate-ingestion

Validate Bronze ingestion artifacts against LLD specifications and report issues.

Official
Advanced
weiflycc-cmdweiflycc-cmd

e2e-medallion-architecture

Implement end-to-end medallion architecture lakehouse patterns in Microsoft Fabric.

Community
Advanced
patrikboroschpatrikborosch
14

hdinsight-migration

Migrate HDInsight Spark, Hive, and Oozie workloads to Microsoft Fabric.

Community
Advanced
heymishyheymishy

coverage-map

Read test plans and pipeline-state.json to report per-story and per-AC coverage status.

Community
Intermediate