data-engineer

To design scalable data pipelines for ETL and ELT, structure processing logic across frameworks like Spark, Kafka, Flink, or Beam. This Skill provides architectural patterns and robust guardrails to build reliable data infrastructure for analytics workloads.

Optimizing data warehouse costs across Snowflake, BigQuery, and Redshift requires applying query tuning and infrastructure governance. This Skill implements cost governance best practices to reduce compute spend and improve performance.

Data orchestration with Airflow, Prefect, or Dagster coordinates automated pipeline execution, dependency management, and data quality enforcement. This Skill helps configure these tools to maintain reliable, auditable data workflows.

Yes, you can use this Skill to build lakehouse architectures integrated with data quality frameworks. It supports designing reliable, auditable data workflows that enforce governance and quality checks across your infrastructure.

Choosing Spark or Flink for data engineering pipelines depends on whether your workloads require batch or stream processing. This Skill guides architecture selection to match the appropriate tool with your pipeline patterns and scalability requirements.