Create idempotent ELT pipelines with Airflow, dbt, Spark, and Kafka.
npx skills add https://github.com/Exia-thd/Digital-Nervous --skill data-engineer-exia-thd
Please help me install this Agent Skill. Skill: data-engineer Source: https://github.com/Exia-thd/Digital-Nervous/tree/main/skills/data-engineer Command: npx skills add https://github.com/Exia-thd/Digital-Nervous --skill data-engineer-exia-thd
Organizations struggle to reliably move, transform, and make data accessible at scale, leading to broken pipelines, data quality issues, and delayed insights.
Ask the data engineer skill to design an ELT pipeline that loads daily sales data from an API into a Snowflake warehouse and creates dbt models for reporting.
High-intent search queries and answers about installing and using this skill.
To build idempotent ELT pipelines, this skill designs workflows using Airflow or Dagster that load raw data and apply a medallion architecture with dbt models to ensure repeatable, reliable results across runs.
Medallion architecture is a data design pattern that organizes data into bronze, silver, and gold layers. This skill applies it to progressively refine raw data into structured, analytics-ready reporting tables within your warehouse.
Yes, this skill supports real-time stream processing by designing ingestion workflows that use Spark or Kafka to capture and process streaming data for immediate analytical use cases.
You can integrate data quality testing by defining schema contracts and anomaly detection checks directly within dbt models, ensuring pipeline outputs meet defined quality standards before reaching gold layers.
The best way to load daily API data into Snowflake is through an automated ELT pipeline that extracts source data, stores it in the warehouse, and uses dbt to materialize structured reporting tables for analytics.
Data pipelines break due to lack of idempotency and schema enforcement. This skill prevents these issues by implementing schema contracts, automated tests, and repeatable processing logic to guarantee reliable data delivery.
clawdataQuery and manage Snowflake warehouses, schemas, and tables via the snow CLI.
keithbinklyBuild dbt semantic models, metrics, and BI tool integrations.
ouakarAutomate end-to-end data pipelines from ingestion to analytics-ready datasets.
CatsofsufferingManage and validate data pipelines and streaming workflows across orchestration and quality tools.
