senior-data-engineer

Build scalable data pipelines with Python, SQL, Spark, and Airflow.

Updated Apr 2, 2026

Search Tags:#data pipeline #data modeling #dbt #data engineering #Airflow #Kafka #ETL/ELT

Ready to run

npx skills add https://github.com/4lerman/text_evaluator --skill senior-data-engineer-4lerman

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/4lerman/text_evaluator/tree/main/.agents/skills/engineering-skills/senior-data-engineer
Command: npx skills add https://github.com/4lerman/text_evaluator --skill senior-data-engineer-4lerman

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, numpy, sqlalchemy, pyarrow, dask, spark, airflow, dbt, kafka, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides expertise in building scalable data pipelines, ETL/ELT systems, and data infrastructure, addressing the challenges of data modeling, pipeline orchestration, and DataOps.

Core Features & Use Cases

Scalable Data Pipelines: Design and build scalable pipelines for data ingestion, transformation, and storage.
ETL/ELT Systems: Develop robust ETL/ELT processes for data integration and transformation.
Data Infrastructure: Build a modern data stack with Airflow, dbt, Kafka, and other tools.
Use Case: Imagine you are working with large volumes of data and need to build a data pipeline that ingests data from multiple sources, performs complex transformations, and loads it into a data warehouse for analysis.

Quick Start

Activate the 'senior-data-engineer' skill and follow the guidance provided to start building your data engineering solution.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build a scalable data pipeline for ingesting and transforming large volumes of data?▼

Build scalable data pipelines by orchestrating data ingestion and transformation using Python, Spark, and Airflow. This approach handles large volumes of data from multiple sources, performs complex transformations, and loads it into a data warehouse for analysis.

What is the difference between ETL and ELT systems when designing data infrastructure?▼

ETL transforms data before loading it into the warehouse, while ELT loads raw data first and transforms it within the warehouse. This skill develops robust ETL and ELT processes using dbt and SQL to handle data integration and transformation efficiently.

Can I use Airflow and dbt together for pipeline orchestration and data modeling?▼

Yes, Airflow and dbt work together for pipeline orchestration and data modeling. Airflow manages scheduling and workflow execution, while dbt handles SQL-based data transformations and modeling within a modern data infrastructure stack.

Do I need Spark and Kafka to process large-scale data architecture tasks?▼

Spark and Kafka are required for large-scale data processing and complex data architecture design. Spark handles distributed data transformations, while Kafka enables real-time data streaming and ingestion across the end-to-end data lifecycle.

What is the best way to manage DataOps and data lifecycle across multiple sources?▼

Manage DataOps and the end-to-end data lifecycle by building a modern data stack with Airflow, dbt, and Kafka. This approach ensures robust pipeline orchestration, scalable data integration, and reliable infrastructure for continuous data delivery.

Why does my data modeling process fail when handling complex transformations in SQL?▼

Data modeling processes fail when SQL transformations lack structured orchestration and scalable infrastructure. Implementing dbt for data modeling alongside Airflow for pipeline orchestration ensures complex transformations are executed reliably and maintainably.