spark-python-data-source

To create a custom PySpark data source, implement the DataSource, DataSourceReader, and DataSourceWriter interfaces using the PySpark DataSource API. This Skill provides a structural blueprint for building batch and streaming connectors to external systems like databases, APIs, and queues.

Yes, you can build both batch and streaming connectors using the PySpark DataSource API. The Skill guides the implementation of DataSourceStreamReader and DataSourceStreamWriter interfaces alongside batch equivalents to enable end-to-end data transfer workflows.

The recommended architecture for Spark data sources uses a flat single-level inheritance structure with in-executor imports. This design minimizes dependencies and ensures reliable connector development across custom protocols and external systems.

Yes, this PySpark data source approach works for ingesting data from REST APIs and message queues. The Skill provides real-world patterns for data ingestion and pushing data to external systems for both batch and streaming workloads.

In-executor imports are required when developing PySpark data source connectors to ensure that necessary dependencies are available directly on the executor nodes. This prevents serialization errors and maintains a flat inheritance model during data transfer workflows.