confluent-kafka-connect

Kafka Connect pipelines use source connectors like JDBC or Debezium CDC to read from databases and write to Kafka topics. Configure the connector with database credentials, table selection, and polling mode (incrementing, timestamp, or timestamp+incrementing), then deploy it to a Kafka Connect cluster to begin continuous data movement.

JDBC polling queries tables at intervals and works for any SQL database, while Debezium CDC reads database transaction logs for true change events. Debezium provides lower latency and captures deletes, but requires database-specific connectors; JDBC is simpler but may miss rapid changes between polls.

Single Message Transforms (SMTs) enable field-level operations including masking sensitive data. Apply SMTs in connector configuration to redact, rename, or filter fields before records reach Kafka, reducing exposure of PII without modifying source systems.

Sink connectors route Kafka topics to external systems; Elasticsearch Sink indexes records for search, S3 Sink archives data to object storage, and JDBC Sink writes back to databases. Configure the sink with target credentials and field mappings to route and transform data at scale.

Task parallelism distributes connector work across multiple workers for higher throughput. Set `tasks.max` in connector configuration; ensure source supports partitioning (JDBC with incrementing columns, Debezium partitions) and sink can handle concurrent writes to avoid duplicate or out-of-order records.

Debezium connectors output messages in envelope format containing before/after values and metadata; mismatches with downstream SMTs or sinks cause failures. Validate schema compatibility, confirm Debezium envelope settings match sink expectations, and use SMTs to reshape records if needed.