clickzetta-table-stream-pipeline

To build an incremental ETL pipeline, enable change_tracking on the source table, create a TABLE STREAM to capture inserts, updates, and deletions, prepare the target table, and consume changes via MERGE for idempotent synchronization.

STANDARD mode captures all inserts, updates, and deletions for data synchronization, while APPEND_ONLY mode captures only new inserts, making it suitable for historical auditing and log-based data tracking without update overhead.

You can prevent duplicates during incremental ETL by applying offset-based consumption and using MERGE operations, which ensures idempotent data delivery and reliable synchronization even if the pipeline is rerun.

Yes, the pipeline supports data preview functionality, allowing you to inspect captured table stream changes before applying them, which facilitates fault-tolerant processing and ensures accurate downstream delivery.

A resumable CDC pipeline is achieved through offset-based consumption and change_tracking management, allowing the pipeline to restart from the last processed offset without losing data integrity or duplicating records.

Use table stream CDC for near real-time data replication and incremental ETL when source tables change frequently, as it tracks only modifications rather than repeatedly processing entire datasets.