Design ETL/ELT pipelines and data warehouses with Airflow, dbt, Kafka, and Spark.
npx skills add https://github.com/grasberg/sofia --skill data-engineer-grasberg
Please help me install this Agent Skill. Skill: data-engineer Source: https://github.com/grasberg/sofia/tree/main/workspace/skills/data-engineer Command: npx skills add https://github.com/grasberg/sofia --skill data-engineer-grasberg
Data engineers often design and maintain complex data pipelines, warehouses, and governance systems, which are error-prone and hard to scale.
Design a scalable ETL/ELT pipeline for a new dataset, including source selection, schema design, orchestration, and quality checks.
High-intent search queries and answers about installing and using this skill.
To design a scalable ETL pipeline, you must select appropriate data sources, define schema evolution rules, configure orchestration with tools like Airflow, and implement data quality checks to ensure reliable batch loading.
Handling schema evolution requires enforcing data quality gates and validation rules within your pipeline. This prevents downstream breakages by ensuring that changes to source schemas are monitored and managed before loading into the data warehouse.
Idempotent loading ensures that reprocessing batch or streaming data yields the same results without duplication. This is achieved by designing pipelines that track processing state and handle retries safely using tools like Kafka and Spark.
Yes, you can use Airflow to orchestrate ELT workflows and dbt to manage transformations and enforce data quality. Together they provide reliable scheduling, monitoring, and schema validation for data warehouses.
You need streaming data pipelines when real-time data ingestion is required for immediate processing. Tools like Kafka and Spark enable continuous data flow and monitoring, whereas batch processing handles scheduled, high-volume workloads efficiently.
AngelP17Automate end-to-end ML pipelines from data ingestion to deployment.
nguyenpv1980-wqDesign warehouse, lake, or lakehouse architecture with governed zones and tenant isolation.
SSiertsemaDefine data governance policies covering stewardship, quality, access, lifecycle, lineage, and metadata.
ai-enhanced-engineerAutomate CI/CD pipeline design on Google Cloud Platform with Cloud Build and Cloud Deploy.
