data-pipeline-engineer

To build a batch ETL pipeline for a star schema, you can use this Skill to design an Airflow DAG that orchestrates data ingestion from a transactional database, applies transformations, and ensures data accuracy at each stage.

The best way to orchestrate data pipelines with Airflow and dbt is to use this Skill to generate integration patterns that schedule workflows, execute transformations, and run dbt tests for data validation.

To choose between Medallion, Lambda, and Kappa architectures for data warehousing, this Skill guides you through selecting the right streaming and batch processing architecture based on your specific production requirements.

Yes, you can use Spark and Kafka together for stream processing and batch processing, as this Skill provides technology integration patterns for building robust streaming architectures and handling both processing methods.

To integrate Great Expectations for data quality checks in an ELT pipeline, this Skill helps you incorporate validation frameworks alongside dbt tests to prevent common anti-patterns and ensure data accuracy.

Common data pipeline anti-patterns to avoid in production include inadequate data quality checks and poor orchestration, which this Skill addresses by providing patterns for Great Expectations, dbt tests, and robust architecture design.