implement-partitioned-connector

Partitioned data connectors are implemented by extending the LakeflowConnect interfaces. Defining methods like get_partitions and latest_offset allows Spark to efficiently process new streaming data in parallel micro-batches.

Batch partitioned reads use the SupportsPartition interface, while streaming partitioned reads use SupportsPartitionedStream. Both enable parallel data ingestion, but streaming continuously tracks new data using the latest_offset method.

Use SupportsPartitionedStream when developing a streaming connector for naturally partitioned sources like time-series databases. It enables Spark to efficiently process new data in micro-batches by defining get_partitions and latest_offset methods.

Set up parallel processing by implementing a partitioned connector that defines get_partitions and read_partition methods. This allows Spark to efficiently ingest time-range partitioned data from the database in parallel.

Yes, LakeflowConnect supports both batch and streaming partitioned data ingestion. Connectors adhere to interface conformance by implementing SupportsPartition for batch processing or SupportsPartitionedStream for streaming workflows.