data-eng-pipeline-architect

Design a production data pipeline by mapping batch and streaming workloads to a Medallion lakehouse architecture, using tools like Airflow, Spark, and Kafka to orchestrate data flows and enforce data quality checks across Bronze, Silver, and Gold layers.

The Medallion architecture pattern organizes data warehouse modeling into Bronze, Silver, and Gold layers, progressively refining raw data into curated analytics-ready datasets using dbt and modern data platforms like Snowflake, BigQuery, or Databricks.

Implement CI/CD and data quality tests in an ETL pipeline by using dbt to define data models, apply schema tests, and enforce DataOps practices, ensuring pipeline reliability and observability across your data infrastructure.

Yes, you can use Airflow and Kafka together for pipeline orchestration and streaming ingestion, coordinating batch processing with Spark while Kafka handles real-time streaming data feeds into your data infrastructure.

The best way to enforce data quality and observability in an ELT pipeline is by applying DataOps practices, integrating automated quality checks, monitoring, and testing frameworks within your dbt and orchestration workflows.

Choose a star schema for simpler, denormalized data modeling optimized for querying, or a snowflake schema when normalizing dimensions saves storage and fits your platform's specific querying and cost constraints.