senior-data-engineer

Automate scalable data pipeline design and warehousing with Spark, Airflow, and Snowflake.

467|103|Updated Jan 13, 2026

Search Tags:#data #streaming #etl #airflow #spark #data-pipelines #snowflake

Ready to run

npx skills add https://github.com/borghei/Claude-Skills --skill senior-data-engineer-borghei

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/borghei/Claude-Skills/tree/main/engineering/senior-data-engineer
Command: npx skills add https://github.com/borghei/Claude-Skills --skill senior-data-engineer-borghei

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Modern analytics teams struggle to design, implement, and maintain scalable data pipelines and data warehouses, leading to delays and data quality issues.

Core Features & Use Cases

Batch and streaming ETL/ELT design for scalable data pipelines.
Data warehousing architecture, modeling, and governance for analytics-ready datasets.
End-to-end workflows spanning ingestion, processing with Spark, orchestration with Airflow, and cloud platforms like Snowflake.
Use Case: Build an end-to-end pipeline from source systems to an analytics dashboard with automated data quality checks and lineage.

Quick Start

Define a minimal end-to-end pipeline skeleton that ingests data, processes it with Spark, and loads it into Snowflake, then orchestrate with Airflow.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build scalable data pipelines for batch and streaming ETL workflows?▼

Scalable data pipelines for batch and streaming ETL workflows are designed by orchestrating ingestion, processing, and warehousing. You can automate this architecture using Spark for processing, Airflow for orchestration, and Snowflake for warehousing to enable end-to-end analytics.

What is the best way to architect a data warehouse for analytics-ready datasets?▼

Architecting a data warehouse for analytics-ready datasets involves structured data modeling, governance, and quality checks. This approach automates warehousing design across cloud platforms like Snowflake, ensuring datasets are optimized for modern analytics consumption.

Can I orchestrate Spark and Snowflake ELT pipelines using Airflow?▼

Yes, you can orchestrate Spark and Snowflake ELT pipelines using Airflow. This combination automates end-to-end workflows spanning data ingestion, processing with Spark, orchestration with Airflow, and loading into Snowflake for analytics.

How do I add automated data quality checks and lineage to an ETL pipeline?▼

Automated data quality checks and lineage are added to an ETL pipeline by designing end-to-end workflows from source systems to analytics dashboards. This requires Spark for processing and Airflow for orchestration to enforce governance across the pipeline.

Do I need Spark, Airflow, and Snowflake to enable end-to-end analytics pipelines?▼

Yes, you need Spark, Airflow, and Snowflake to enable end-to-end analytics pipelines. Spark is required for data processing, Airflow for pipeline orchestration, and Snowflake for cloud warehousing to fully automate scalable batch and streaming workflows.