data-eng-pipeline-architect

Design production-grade data pipelines for batch and streaming workloads.

Updated Jan 28, 2026

Search Tags:#dbt #etl #airflow #kafka #dataops #spark #data-pipeline

Ready to run

npx skills add https://github.com/scanady/nexus-agents --skill data-eng-pipeline-architect

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-eng-pipeline-architect
Source: https://github.com/scanady/nexus-agents/tree/main/skills/data-eng-pipeline-architect
Command: npx skills add https://github.com/scanady/nexus-agents --skill data-eng-pipeline-architect

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Design and build production-grade data pipelines and data infrastructure using modern tools and DataOps practices.

Core Features & Use Cases

Design architecture for batch and streaming workloads, including Bronze/Silver/Gold patterns and Medallion lakehouse concepts.
Model data and implement CI/CD for data pipelines with quality checks, tests, and observability.
Use cases include building end-to-end ETL/ELT, data quality enforcement, and data warehouse modeling across Snowflake, BigQuery, or Databricks.

Quick Start

Draft a production-grade data pipeline design for a batch and streaming workload using your current toolset.

Frequently Asked Questions about data-eng-pipeline-architect

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design a production data pipeline for both batch and streaming workloads?▼

Design a production data pipeline by mapping batch and streaming workloads to a Medallion lakehouse architecture, using tools like Airflow, Spark, and Kafka to orchestrate data flows and enforce data quality checks across Bronze, Silver, and Gold layers.

What is the Medallion architecture pattern for data warehouse modeling?▼

The Medallion architecture pattern organizes data warehouse modeling into Bronze, Silver, and Gold layers, progressively refining raw data into curated analytics-ready datasets using dbt and modern data platforms like Snowflake, BigQuery, or Databricks.

How do I implement CI/CD and data quality tests in an ETL pipeline with dbt?▼

Implement CI/CD and data quality tests in an ETL pipeline by using dbt to define data models, apply schema tests, and enforce DataOps practices, ensuring pipeline reliability and observability across your data infrastructure.

Can I use Airflow and Kafka together for pipeline orchestration and streaming ingestion?▼

Yes, you can use Airflow and Kafka together for pipeline orchestration and streaming ingestion, coordinating batch processing with Spark while Kafka handles real-time streaming data feeds into your data infrastructure.

What is the best way to enforce data quality and observability in an ELT pipeline?▼

The best way to enforce data quality and observability in an ELT pipeline is by applying DataOps practices, integrating automated quality checks, monitoring, and testing frameworks within your dbt and orchestration workflows.

When should I choose a star schema versus snowflake schema for data modeling?▼

Choose a star schema for simpler, denormalized data modeling optimized for querying, or a snowflake schema when normalizing dimensions saves storage and fits your platform's specific querying and cost constraints.