confluent-kafka-connect

Configure and manage Kafka Connect source and sink connectors.

156|21|Updated Oct 25, 2025

Search Tags:#data-pipelines #jdbc #kafka-connect #smt #debezium #confluent

Ready to run

npx skills add https://github.com/anton-abyzov/specweave --skill confluent-kafka-connect

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: confluent-kafka-connect
Source: https://github.com/anton-abyzov/specweave/tree/main/plugins/specweave-confluent/skills/confluent-kafka-connect
Command: npx skills add https://github.com/anton-abyzov/specweave --skill confluent-kafka-connect

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps teams design, configure, and troubleshoot Kafka Connect pipelines, enabling seamless movement of data from source systems to sinks with minimal manual setup.

Core Features & Use Cases

Source Connectors (External System → Kafka): JDBC Source, Debezium CDC, S3 Source, File Source
Sink Connectors (Kafka → External System): JDBC Sink, Elasticsearch Sink, S3 Sink, HDFS Sink
Single Message Transforms (SMTs): Field operations, routing, filtering
Patterns & Troubleshooting: Connector configuration best practices, common failure modes, and optimization

Quick Start

Configure a JDBC source to stream a database table into a Kafka topic named 'db-orders' with Debezium CDC for change data capture, then apply a simple SMT to mask sensitive fields.

Frequently Asked Questions about confluent-kafka-connect

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up a Kafka Connect pipeline to stream data from a database?▼

Kafka Connect pipelines use source connectors like JDBC or Debezium CDC to read from databases and write to Kafka topics. Configure the connector with database credentials, table selection, and polling mode (incrementing, timestamp, or timestamp+incrementing), then deploy it to a Kafka Connect cluster to begin continuous data movement.

What's the difference between JDBC and Debezium for capturing database changes?▼

JDBC polling queries tables at intervals and works for any SQL database, while Debezium CDC reads database transaction logs for true change events. Debezium provides lower latency and captures deletes, but requires database-specific connectors; JDBC is simpler but may miss rapid changes between polls.

Can I mask sensitive fields in Kafka Connect pipelines?▼

Single Message Transforms (SMTs) enable field-level operations including masking sensitive data. Apply SMTs in connector configuration to redact, rename, or filter fields before records reach Kafka, reducing exposure of PII without modifying source systems.

How do I stream Kafka data into Elasticsearch or S3?▼

Sink connectors route Kafka topics to external systems; Elasticsearch Sink indexes records for search, S3 Sink archives data to object storage, and JDBC Sink writes back to databases. Configure the sink with target credentials and field mappings to route and transform data at scale.

What configurations do I need for parallel Kafka Connect tasks?▼

Task parallelism distributes connector work across multiple workers for higher throughput. Set `tasks.max` in connector configuration; ensure source supports partitioning (JDBC with incrementing columns, Debezium partitions) and sink can handle concurrent writes to avoid duplicate or out-of-order records.

Why does my Kafka Connect connector fail with schema or envelope errors?▼

Debezium connectors output messages in envelope format containing before/after values and metadata; mismatches with downstream SMTs or sinks cause failures. Validate schema compatibility, confirm Debezium envelope settings match sink expectations, and use SMTs to reshape records if needed.