skills.rest
singhaganesh
singhaganeshCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

senior-data-engineer

Designs data pipelines, ETL workflows, and data quality validation using Python, SQL, Spark, and Airflow.

Updated Jul 7, 2026
Search Tags:#data quality#data engineering#etl#airflow#dataops#spark#pipeline orchestration
One-click install
npx skills add https://github.com/singhaganesh/Urban-assist --skill senior-data-engineer-singhaganesh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/singhaganesh/Urban-assist/tree/main/.cursor/skills/senior-data-engineer
Command: npx skills add https://github.com/singhaganesh/Urban-assist --skill senior-data-engineer-singhaganesh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Building scalable data pipelines, ETL/ELT systems, and reliable data infrastructure requires deep expertise across orchestration, data modeling, quality validation, and DataOps, which this Skill consolidates into one guided workflow. ## Core Features & Use Cases - Pipeline Orchestration: Provides a pipeline_orchestrator.py script and architecture references for designing batch and real-time data workflows with Airflow, Kafka, and Spark. - Data Quality Validation: Includes a data_quality_validator.py script and best-practice references for enforcing schema checks, reliability targets, and observability. - ETL Performance Optimization: Ships an etl_performance_optimizer.py script plus guidance on distributed processing, caching, and cost optimization. - Use Case: When designing a new analytics platform, use this Skill to model the warehouse schema, orchestrate ingestion pipelines, and validate data quality before production deployment. ## Quick Start Ask the assistant to design a scalable ETL pipeline with data quality checks for your project using this data engineering skill.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate a data pipeline with Python?▼

Run the pipeline_orchestrator.py script with --input and --output arguments to execute a pipeline workflow. The skill's references also cover Airflow-based orchestration patterns for scheduling and monitoring production pipelines.

How to validate data quality in an ETL pipeline?▼

Use the data_quality_validator.py script with input and output paths to run validation checks on your datasets. The dataops_best_practices.md reference covers reliability targets, monitoring, and failure-handling strategies.

What tools does this skill cover for data engineering?▼

It covers Python, SQL, Spark, Airflow, dbt, Kafka, and Databricks for pipelines, plus PostgreSQL, BigQuery, and Snowflake for storage. Deployment guidance includes Docker, Kubernetes, and AWS/GCP/Azure.

Can I use this skill for real-time streaming data?▼

Yes, the references describe real-time processing patterns including Kafka-based streaming, batching and caching strategies, and latency optimization targets such as P99 under 200ms.

What are the limitations of the included Python scripts?▼

The scripts are structured templates with placeholder execution logic, so you must implement the core processing in the _execute method. They provide CLI argument parsing, logging, and error handling scaffolding rather than complete implementations.

Related Skills

Explore More Data & Analytics Skills→
ancolemanancoleman
503

ingesting-data

Facilitates data ingestion from cloud, APIs, files, and streaming sources into databases.

Community
Advanced
LordjiggyLordjiggy

data-engineer

Automate ETL pipeline design, orchestration, and data warehouse schema modeling.

Community
Intermediate
timequitytimequity
6

data-governance

Enforce data governance across data pipelines with lineage tracking and access controls.

Official
Advanced
4lerman4lerman

senior-data-engineer

Build scalable data pipelines with Python, SQL, Spark, and Airflow.

Community
Advanced