data-engineering

To review data engineering pull requests, compare main..HEAD changes to assess the diff for contract breaks, idempotency, backfill safety, observability gaps, and security/PII risks. This structured checklist ensures correctness and safe operations across pipeline changes.

Designing streaming pipelines with data contracts requires defining explicit schema, semantics, and SLAs, alongside processing choices like watermark or offset tracking. This approach guides inputs, quality gates, and operational runbooks for lakehouse and warehouse workflows.

Planning safe backfills and replays for ETL pipelines requires ensuring idempotency and cost-aware incremental processing. The design workflow evaluates operational runbooks, data loss or duplication risks, and replay strategies to maintain production safety during pipeline updates.

Yes, the data pipeline review process works with Confluent Kafka and Teradata, alongside Databricks and Snowflake. It applies platform-specific rule pointers to target the most relevant guidance for ETL/ELT and streaming workflows across these specific environments.

Observability instrumentation and governance controls in ETL workflows are needed to satisfy production operations and least privilege requirements. Applying these controls during pipeline design prevents security or PII exposure and ensures continuous operational monitoring.

Batch processing pipelines lack contract-driven semantics and quality gates when design workflows skip explicit schema, SLA, and semantic definitions. Implementing these gates ensures correctness, prevents data loss, and validates processing choices before lakehouse deployment.