senior-data-engineer

Design and implement scalable data pipelines with Airflow, dbt, Spark, and Kafka.

Updated Dec 23, 2024
One-click install
npx skills add https://github.com/salamientark/dotfiles --skill senior-data-engineer-salamientark
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/salamientark/dotfiles/tree/main/claude/skills/engineering-team/senior-data-engineer
Command: npx skills add https://github.com/salamientark/dotfiles --skill senior-data-engineer-salamientark

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Data teams often struggle to design scalable data architectures, implement robust ETL/ELT pipelines, enforce data quality and governance, and maintain reproducible data workflows across multi-source environments.

Core Features & Use Cases

  • Production-grade pipeline design and orchestration across Airflow, dbt, Spark, and streaming platforms to handle complex end-to-end data workflows.
  • Data quality, lineage, and governance patterns including data contracts, validation, monitoring, and DataOps practices to ensure reliability and compliance.
  • Architecture decision guidance (Batch vs Streaming, Lakehouse vs Warehouse, and SCD/ DV patterns) with practical examples for modern data stacks.

Quick Start

Initialize a project to design and deploy a scalable data platform, including a sample pipeline, data contracts, and governance checks.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design scalable data pipelines and architecture using Airflow and dbt?

Design scalable data pipelines by orchestrating ETL/ELT workflows with Airflow and dbt, applying production-grade patterns for data quality, monitoring, and lineage. This approach handles complex end-to-end data workflows across multi-source environments.

What is the best way to enforce data quality and governance in enterprise data platforms?

Enforce data quality and governance by implementing data contracts, validation checks, and DataOps practices. This ensures reliability and compliance across your data infrastructure through codified monitoring and lineage patterns.

How do I choose between batch and streaming or Lakehouse and Warehouse architectures for my data stack?

Choose between batch and streaming or Lakehouse and Warehouse architectures using provided decision guidance. It offers practical examples and architecture patterns like SCD and Data Vault to guide modern data stack implementations.

Can I use Python, SQL, and Spark together for end-to-end ETL workflows?

Yes, you can use Python, SQL, and Spark together for ETL workflows. The design supports multi-source environments, integrating these technologies with Airflow, dbt, and Kafka to handle complex data engineering tasks.

How do I initialize a project to deploy a scalable data platform with governance checks?

Initialize a project through a quick start process that sets up a scalable data platform. This includes deploying a sample pipeline, defining data contracts, and configuring governance checks for immediate DataOps implementation.