clickzetta-kafka-ingest-pipeline

What problem does it solve?

Ingesting Kafka data into ClickZetta Lakehouse end-to-end is complex and error-prone. This Skill streamlines the process by providing a structured workflow to validate connections, parse messages, create target schemas, and manage continuous loading via configurable pipes and tables.

Core Features & Use Cases

• Two ingestion paths: READ_KAFKA Pipe for streaming loads and Kafka external table + Table Stream for raw message retention and multiple downstream consumers.
• End-to-end workflow: validates connectivity, probes JSON payloads, creates/updates target tables, and deploys pipelines with batch tuning and offset controls.
• Use Case: Ingest orders events in near real-time from Kafka into Lakehouse, with optional SASL authentication and offset reset options.

Quick Start

Create a READ_KAFKA Pipe to start ingesting JSON messages from a Kafka topic into a Lakehouse table.