Translate DES source requirements into dlt ingestion pipelines with evidence.
npx skills add https://github.com/DKSang/DES-SKILL --skill des-dlt-ingestion
Please help me install this Agent Skill. Skill: des-dlt-ingestion Source: https://github.com/DKSang/DES-SKILL/tree/main/skills-stack/des-dlt-ingestion Command: npx skills add https://github.com/DKSang/DES-SKILL --skill des-dlt-ingestion
It helps you design and implement dependable dlt ingestion pipelines that load DES Bronze datasets with correct write behavior, incremental state, schema evolution handling, and evidence for downstream phases.
Use the des-dlt-ingestion skill to generate a dlt ingestion plan and evidence for your REST API or SQL database source, aligned to your DES Phase 08 ingestion specification and Bronze layout.
High-intent search queries and answers about installing and using this skill.
Schema evolution policies in dlt ingestion control how the pipeline reacts to schema drift, capturing new columns, type changes, and nested structure modifications while enforcing data contracts and outputting standardized evidence for downstream handoff.
dlt ingestion pipelines handle schema drift by documenting contract behavior for new columns, type changes, and nested structures, capturing resulting evidence to enforce data contracts during Bronze layer loading.
dlt ingestion pipelines support REST APIs, SQL databases, and filesystem or cloud storage sources, translating DES-defined source specifications into executable loading plans with traceable pipeline evidence across multiple ingestion phases.
Incremental loading in dlt ingestion manages cursor state, lag windows, and backfill strategies to control append, replace, or merge write dispositions, ensuring dependable Bronze dataset loads from REST API or SQL database sources.
To configure write disposition and merge strategies in dlt pipelines, select append, replace, or merge modes based on source assessment, then define incremental cursor state, backfill strategies, and dataset naming for Bronze layer loads.
dlt ingestion generates pipeline evidence artifacts by producing standardized load records during Bronze dataset ingestion, documenting write disposition, incremental state, and schema evolution behavior for downstream handoff to DuckDB, dbt, Fabric, Provero, or Airflow.
prashsubOrchestrate Silver layer pipeline creation with Spark Declarative Pipelines and data quality rules.
DKSangOrchestrate DES workflows into scheduled Apache Airflow DAGs with validation gates.
Sheldon-92Apply governance rules to standardize AI data pipelines.
erichowensAutomate ETL/ELT pipeline design, validation, and operation across medallion architectures.
