skills.rest
ricardolui
ricardoluiCOMMUNITY
Explore all skills by this author →
GitHub

confluent-cloud-cdc-tableflow

Configure end-to-end CDC pipelines from databases to Iceberg or Delta tables via Confluent Cloud.

Updated Jul 7, 2026
Search Tags:#iceberg#cdc#debezium#flink#delta lake#confluent cloud#tableflow
One-click install
npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill confluent-cloud-cdc-tableflow-ricardolui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: confluent-cloud-cdc-tableflow
Source: https://github.com/ricardolui/gcp-custom-agent-skills/tree/main/confluent-cloud-cdc-tableflow
Command: npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill confluent-cloud-cdc-tableflow-ricardolui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Setting up a Change Data Capture pipeline from a database to a data lake involves coordinating Debezium connectors, Schema Registry, Flink SQL, and Tableflow, with many failure-prone details like tombstone handling, changelog modes, and type mappings. This Skill guides the full workflow so database changes land correctly in Iceberg or Delta Lake tables. ## Core Features & Use Cases - Connector Configuration: Generates ready-to-use configs for PostgreSQL, MySQL, SQL Server, Oracle XStream, and DynamoDB CDC source connectors with JSON_SR, Avro, or Protobuf formats. - Flink Decode Layer: Creates target tables with upsert changelog mode and INSERT statements that decode Debezium envelopes, including handling for schemaless and multi-event topics. - Tableflow Enablement: Enables Tableflow on target topics for Iceberg or Delta Lake with managed or BYOB storage, plus verification and troubleshooting guidance. - Use Case: A team wants to replicate a PostgreSQL 'orders' table into an Iceberg table in S3. The Skill produces the connector config, Flink SQL, and Tableflow setup, then walks through validation and testing with a live insert. ## Quick Start Ask the assistant to set up a CDC pipeline from your database to an Iceberg or Delta table on Confluent Cloud, providing your database type, connection details, tables to capture, and target format.

Frequently Asked Questions about confluent-cloud-cdc-tableflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up a CDC pipeline from PostgreSQL to Iceberg on Confluent Cloud?▼

Create a PostgreSQL CDC Source V2 connector with a unique topic.prefix, then use Flink to decode the Debezium envelope into a target topic with changelog.mode set to upsert, and finally enable Tableflow on that target topic for Iceberg. Never enable Tableflow directly on the CDC source topic because tombstone records suspend it.

Why does Tableflow suspend with a null value Kafka record error?▼

Tableflow suspends because CDC connectors with tombstones.on.delete=true produce null-value records on DELETE operations, which APPEND-mode Tableflow cannot handle. The fix is to route the CDC topic through a Flink INSERT into an upsert-mode target topic before enabling Tableflow.

Which schema format should I use for Debezium CDC connectors: JSON_SR, Avro, or Protobuf?▼

All three formats register schemas in Schema Registry and work identically with Flink auto-discovery and Tableflow. JSON_SR is human-readable and easiest to debug, while Avro and Protobuf produce roughly 30-50% smaller payloads for high-throughput pipelines. Never use plain JSON, since it breaks Flink discovery and Tableflow.

Can Tableflow work with a Kafka topic that has no Schema Registry schema?▼

No, Tableflow and Flink auto-discovery require a registered schema. You can register a partial JSON schema manually, use Confluent Cloud schema inference from existing messages, or parse the topic in Flink using raw BYTES with JSON_VALUE functions.

How do I change the Tableflow changelog mode after it is already materialized?▼

The changelog mode is immutable once Tableflow first materializes data, and attempting to change it corrupts state. You must delete the Tableflow topic, delete the underlying Kafka topic, and recreate both from scratch with the desired mode.

Why does DynamoDB CDC snapshot work but streaming silently fails?▼

The CDC phase requires IAM write permissions for a KCL-style checkpointing table that tracks shard leases. Without CreateTable, PutItem, GetItem, UpdateItem, and DeleteItem permissions, the snapshot succeeds but no real-time changes are captured.

Related Skills

Explore More Data & Analytics Skills→
confluentincconfluentinc
42

confluent-cloud-cdc-tableflow

Configure CDC pipelines on Confluent Cloud with Flink SQL and Tableflow sinks.

Official
Advanced
awsaws
2.3k

creating-data-lake-table

Create Iceberg tables in Amazon S3 Tables with schemas and partitioning.

Official
Advanced
gemini-cli-extensionsgemini-cli-extensions
42

cloud-sql-postgres-replication

Monitor Cloud SQL for PostgreSQL replication lag and role permissions.

Official
Intermediate
AbsolutelySkilledAbsolutelySkilled
207

real-time-streaming

Guide building and operating Kafka and Flink streaming pipelines with exactly-once semantics.

Official
Advanced