principal-data-engineer

To design a scalable data platform with Iceberg and Airflow, you need to enforce data contracts, implement idempotent pipelines, and establish OpenLineage tracking. This Skill provides architectural patterns for decoupling systems and ensuring lakehouse reliability across batch and streaming contexts.

Essential lakehouse observability standards require data contracts, OpenLineage integration, and rigorous data quality checks. This Skill specifies SLA monitoring and testing regimes including unit, integration, and data quality tests to maintain platform reliability and maintainability.

Yes, this guidance establishes dbt documentation and comprehensive testing regimes for pipeline engineering. It defines Airflow and Python code standards, specifies anti-patterns to avoid, and sets quality gates to ensure your data transformations meet enterprise governance requirements.

Yes, you can use Polars and DuckDB within a composable data stack. This Skill provides implementation guidance for flexible pipelines using these tools alongside dlt and Iceberg, ensuring your architecture supports both batch and streaming processing efficiently.

The best way to govern data platforms and prevent anti-patterns is through architectural reviews and strict adherence to engineering standards. This Skill guides teams in system decoupling, cost-aware design, and applying Open Data Contract Standards to elevate long-term platform value.