data-pipelines

To design ETL pipelines with idempotency and incremental loading, describe your requirements to generate a plan aligned with the 13-chapter Data Pipelines Pocket Reference. This guidance covers ingestion, storage, and transforms to ensure consistent, repeatable batch processing.

Validating data quality across CDC and streaming pipelines requires applying canonical practices for monitoring and governance. This guidance provides practical templates and checklists to audit validation tasks, ensuring data integrity across real-time and change data capture workflows.

In an ELT pipeline, dbt handles the transformation layer after data lands in storage. The guidance maps orchestration and transformation practices across 13 chapters, ensuring dbt models integrate with staging, monitoring, and incremental loading disciplined workflows.

Yes, Airflow can orchestrate batch and CDC data pipelines together. This guidance provides modeled practices for orchestration, ensuring both batch ETL and CDC workflows maintain idempotency, monitoring, and governance aligned with the 13-chapter reference.

Change data capture is needed instead of standard batch ETL when you require real-time synchronization of source data changes. This guidance helps audit ingestion tasks, applying reference checklists to determine when CDC or streaming fits your pipeline requirements over batch.