senior-data-engineer

Name: senior-data-engineer
Availability: InStock
Author: tiandiyiqi

Design and orchestrate scalable data pipelines with Python, SQL, Spark, and Airflow.

Updated Jan 26, 2026

Search Tags:#scalability #data quality #ETL #data engineering #data pipelines #DataOps

Ready to run

npx skills add https://github.com/tiandiyiqi/ai-skills --skill senior-data-engineer-tiandiyiqi

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/tiandiyiqi/ai-skills/tree/main/engineering-team/senior-data-engineer
Command: npx skills add https://github.com/tiandiyiqi/ai-skills --skill senior-data-engineer-tiandiyiqi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill addresses the complexity of designing, building, and optimizing robust, production-grade data systems and pipelines.

Core Features & Use Cases

Data Pipeline Orchestration: Automate and manage complex data workflows.
Data Quality Validation: Ensure the integrity and accuracy of data.
ETL/ELT Performance Optimization: Tune data processing jobs for efficiency and scalability.
Use Case: A company needs to ingest data from multiple sources, transform it, and load it into a data warehouse for analytics. This skill can orchestrate the entire process, validate data quality at each step, and optimize the performance of the transformations.

Quick Start

Use the senior-data-engineer skill to orchestrate data pipelines starting from the 'data/' directory and outputting to 'results/'.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build scalable data pipelines for ingesting and transforming multiple data sources?▼

To build scalable data pipelines, you orchestrate complex data workflows using tools like Airflow, Spark, and dbt. This involves automating data ingestion, transforming it with Python or SQL, and loading it into a data warehouse for analytics while ensuring data quality.

What's the best way to automate ETL workflows and ensure data quality validation?▼

The best way to automate ETL workflows and validate data quality is by implementing DataOps practices. You orchestrate pipelines using Airflow, apply data quality checks at each transformation step, and optimize performance with Spark or dbt to ensure data integrity.

Does this data engineering approach support modern data stack components like Kafka and dbt?▼

Yes, this data engineering approach fully supports modern data stack components. It integrates Kafka for real-time data streaming, dbt for data modeling and transformations, and Spark for large-scale data processing within your pipeline architecture.

How do I optimize ETL performance and tune data processing jobs for large-scale analytics?▼

To optimize ETL performance, you tune data processing jobs by leveraging Spark for distributed computing and dbt for efficient SQL transformations. This approach scales your data infrastructure, reduces latency, and handles large volumes of data smoothly.

When should I implement DataOps for data pipeline orchestration and data governance?▼

You should implement DataOps when managing complex data workflows that require strict data governance and quality assurance. It automates pipeline orchestration, validates data integrity across ETL systems, and maintains robust production-grade data infrastructure.