clickzetta-data-ingest-pipeline

Data ingestion routing evaluates your source type, latency targets, and data volume against a decision tree to select the optimal pipeline, directing Kafka, OSS, or database sources to specialized ingestion Skills or simple INSERT commands.

For Kafka stream ingestion, the routing mechanism evaluates your real-time latency requirements and volume, then directs the task to the specialized kafka-ingest-pipeline to ensure optimal streaming data flow into the Lakehouse.

Yes, you can sync multiple tables by defining your scope as single table, multiple tables, or full database during routing, which triggers the batch-sync-pipeline or realtime-sync-pipeline based on your CDC latency requirements.

Loading files from OSS or S3 uses the decision tree to assess your batch processing scope, routing the object storage ingestion task to the oss-ingest-pipeline or providing a direct COPY INTO command for simpler file loads.

You should use a specialized ingestion pipeline over a simple INSERT command when dealing with continuous Kafka streams, large-scale OSS batches, or CDC database syncs, whereas simple INSERT or COPY INTO paths suit straightforward, low-volume file loads.

Yes, data ingestion routing supports CDC by evaluating your real-time latency targets and database scope, routing MySQL or PostgreSQL Change Data Capture tasks directly to the realtime-sync-pipeline for continuous synchronization.