connect-cdc-spanner

Stream Google Cloud Spanner CDC mutations into Redpanda or Kafka.

6|3|Updated May 31, 2026

Search Tags:#metadata #kafka #cdc #spanner #connect #redpanda #partition-watermark

One-click install

npx skills add https://github.com/redpanda-data/skills --skill connect-cdc-spanner

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: connect-cdc-spanner
Source: https://github.com/redpanda-data/skills/tree/main/skills/connect-cdc-spanner
Command: npx skills add https://github.com/redpanda-data/skills --skill connect-cdc-spanner

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Streams change data capture (CDC) from Google Cloud Spanner into Redpanda or Kafka using Redpanda Connect's gcp_spanner_cdc input — Spanner change streams, partition-aware watermarked delivery, and metadata persistence. Use when: capturing INSERT/UPDATE/DELETE changes from a Google Cloud Spanner database into Redpanda or Kafka; configuring the gcp_spanner_cdc input; creating a Spanner change stream with CREATE CHANGE STREAM; setting up GCP service-account credentials or Application Default Credentials for the connector; configuring project_id, instance_id, database_id, and stream_id; using start_timestamp or end_timestamp to bound the stream window; understanding the metadata_table the connector creates in Spanner for partition watermarking; filtering mod types with allowed_mod_types (INSERT, UPDATE, DELETE); tuning heartbeat_interval or min_watermark_cache_ttl; understanding the message payload (Mod JSON with keys, new_values, old_values) and message metadata (table_name, mod_type, commit_timestamp, record_sequence, server_transaction_id, transaction_tag); using the batching policy for throughput tuning; the Enterprise license requirement for this connector; routing per-table CDC events to separate Kafka topics with Bloblang; or landing CDC history into Redpanda Enterprise destination features — Iceberg Topics (redpanda.iceberg.mode/delete/ target.lag.ms/partition.spec/invalid.record.action), Tiered Storage (redpanda.remote.write/read, cloud_storage_enabled), Cloud Topics (redpanda.cloud_topic.enabled / redpanda.storage.mode=cloud, cloud_topics_enabled), Remote Read Replicas (redpanda.remote.readreplica), Shadowing for cross-cluster disaster recovery (rpk shadow), and the Redpanda Connect enterprise capabilities (secrets management, the redpanda config-service block, allow/deny lists, FIPS, plus RBAC, OIDC/OAUTHBEARER, Kerberos, Audit Logging, and server-side Schema ID Validation on the destination cluster). All of these require a Redpanda Enterprise license.

Core Features & Use Cases

High-throughput CDC ingestion from Google Cloud Spanner via gcp_spanner_cdc
Per-table change routing and metadata watermarking for resumable streams
Enterprise-license gating with secrets management and advanced destination features
Bi-directional readiness for Redpanda Connect pipelines with configurable time windows

Quick Start

Configure the gcp_spanner_cdc input with your project, instance, database, and stream_id, supply credentials if needed, and start the Redpanda Connect pipeline to begin streaming CDC events.

Frequently Asked Questions about connect-cdc-spanner

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I stream change data capture from Google Cloud Spanner to Kafka?▼

You stream Spanner CDC to Kafka by configuring the gcp_spanner_cdc input in a Redpanda Connect pipeline, which captures INSERT, UPDATE, and DELETE mutations and routes them to your topics.

How does the connector track partition watermarks for resumable Spanner CDC streams?▼

Partition watermarks are tracked using a metadata table that the connector automatically creates within your Spanner database, ensuring streams can resume without data loss after interruptions.

Do I need a Redpanda Enterprise license to use the gcp_spanner_cdc input?▼

Yes, the gcp_spanner_cdc input requires a Redpanda Enterprise license, which also unlocks destination features like Iceberg Topics, Tiered Storage, Cloud Topics, and secrets management.

Can I filter Spanner change stream events to only capture UPDATE and DELETE mutations?▼

Yes, you can filter change stream events by configuring the allowed_mod_types parameter to explicitly include or exclude INSERT, UPDATE, and DELETE mutations from your pipeline.

How do I route per-table Spanner CDC events to separate Kafka topics?▼

You route per-table CDC events to separate topics by applying Bloblang mappings within your Redpanda Connect pipeline, using the table_name metadata field from the Mod JSON payload.

What metadata is included in the payload when capturing Spanner change streams?▼

The payload includes a Mod JSON with keys, new_values, and old_values, alongside metadata containing table_name, mod_type, commit_timestamp, record_sequence, and server_transaction_id.