data-engineer-agent

To build an ETL pipeline, you design and implement data ingestion, transform the data using Spark, and load it into a data warehouse like Snowflake or AWS Redshift. This Skill automates that workflow, including validation and data quality checks.

Implementing data quality checks in a Spark data pipeline involves applying validation frameworks to ensure data integrity during transformation. This Skill designs and integrates these checks directly into your ETL/ELT processes to maintain accuracy.

Yes, you can use Airflow and Kafka to design streaming data architectures. This Skill specializes in creating streaming pipelines with Kafka and orchestrating batch processing workflows with Airflow for scalable data engineering.

Yes, this supports data warehouse modeling with star schemas for cloud platforms. It designs and models data warehouses specifically for cloud environments like Snowflake, Databricks, and AWS Redshift to structure analytics data efficiently.

To automate a daily batch processing workflow for sales data, you configure a scheduled pipeline that extracts data from an API, transforms it with Spark, and loads it into your data warehouse. This Skill designs and validates these automated batch workflows.