clickzetta-kafka-ingest-pipeline

Ingesting Kafka data into a Lakehouse end-to-end is streamlined by validating connectivity, probing JSON payloads, creating target schemas, and deploying continuous loading pipelines with configurable batch sizes and offset controls.

Parsing JSON messages from Kafka into Lakehouse tables is achieved through a structured workflow that probes payloads, creates or updates target schemas, and deploys pipelines using READ_KAFKA Pipe or Kafka external tables with Table Stream.

Yes, this Kafka ingestion workflow supports optional SASL authentication and offset reset options, ensuring secure connections and flexible consumption controls for near real-time streaming scenarios into the Lakehouse.

Use READ_KAFKA Pipe for direct streaming loads into the Lakehouse, whereas Kafka external tables with Table Stream are ideal for retaining raw messages and supporting multiple downstream consumers.

Batched loading for near real-time Kafka streaming works by deploying pipelines with configurable batch sizes and pause controls, orchestrating Vcluster operations to manage continuous data ingestion into the Lakehouse efficiently.