implement-partitioned-connector

Implement partitioned data connectors extending LakeflowConnect with SupportsPartitionedStream or SupportsPartition interfaces.

60|116|Updated Nov 13, 2025

Search Tags:#streaming #data ingestion #partitioning #batch #connector #spark

Ready to run

npx skills add https://github.com/databrickslabs/lakeflow-community-connectors --skill implement-partitioned-connector

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: implement-partitioned-connector
Source: https://github.com/databrickslabs/lakeflow-community-connectors/tree/main/.claude/skills/implement-partitioned-connector
Command: npx skills add https://github.com/databrickslabs/lakeflow-community-connectors --skill implement-partitioned-connector

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the implementation of complex data connectors that support both batch and streaming partitioned reads, ensuring efficient data ingestion from various sources.

Core Features & Use Cases

Partitioned Reads: Implements SupportsPartition for batch and SupportsPartitionedStream for streaming partitioned data.
Interface Conformance: Ensures connectors adhere to LakeflowConnect and partitioning interfaces.
Use Case: Develop a streaming connector for a time-series database where data is naturally partitioned by time ranges, allowing Spark to efficiently process new data in micro-batches.

Quick Start

Implement a partitioned connector for the 'my_source' data source.

Frequently Asked Questions about implement-partitioned-connector

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I implement partitioned data connectors for Spark streaming ingestion?▼

Partitioned data connectors are implemented by extending the LakeflowConnect interfaces. Defining methods like get_partitions and latest_offset allows Spark to efficiently process new streaming data in parallel micro-batches.

What is the difference between batch and streaming partitioned reads in data ingestion?▼

Batch partitioned reads use the SupportsPartition interface, while streaming partitioned reads use SupportsPartitionedStream. Both enable parallel data ingestion, but streaming continuously tracks new data using the latest_offset method.

When do I need to use SupportsPartitionedStream for data ingestion?▼

Use SupportsPartitionedStream when developing a streaming connector for naturally partitioned sources like time-series databases. It enables Spark to efficiently process new data in micro-batches by defining get_partitions and latest_offset methods.

How do I set up parallel processing for a time-series database connector in Spark?▼

Set up parallel processing by implementing a partitioned connector that defines get_partitions and read_partition methods. This allows Spark to efficiently ingest time-range partitioned data from the database in parallel.

Does LakeflowConnect support both batch and streaming partitioned data ingestion?▼

Yes, LakeflowConnect supports both batch and streaming partitioned data ingestion. Connectors adhere to interface conformance by implementing SupportsPartition for batch processing or SupportsPartitionedStream for streaming workflows.