spark-python-data-source

Name: spark-python-data-source
Availability: InStock
Author: aaronachermann

Create PySpark data sources for batch and streaming connectors.

Updated Apr 18, 2026

Search Tags:#streaming #batch #connector #spark #pyspark #datasource

Ready to run

npx skills add https://github.com/aaronachermann/PolentaEncoders --skill spark-python-data-source-aaronachermann

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/aaronachermann/PolentaEncoders/tree/main/.github/skills/spark-python-data-source
Command: npx skills add https://github.com/aaronachermann/PolentaEncoders --skill spark-python-data-source-aaronachermann

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Spark data source development is complex; this Skill provides a guided blueprint to build Python-based DataSource, DataSourceReader/Writer, and DataSourceStreamReader/Writer implementations to connect Spark with external systems (databases, APIs, queues) for batch and streaming workloads.

Core Features & Use Cases

Architectural blueprint for a flat single-level inheritance data source structure and in-executor imports.
Guidance to build connectors, plugins, and custom protocols using PySpark DataSource API for batch and streaming.
Real-world patterns: data ingestion from REST APIs, database sources, and message queues; pushing data to external systems.

Quick Start

Create a new Spark data source project implementing DataSource, DataSourceReader, DataSourceWriter, DataSourceStreamReader, and DataSourceStreamWriter for your target external system.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create a custom PySpark data source to connect Spark with external systems?▼

To create a custom PySpark data source, implement the DataSource, DataSourceReader, and DataSourceWriter interfaces using the PySpark DataSource API. This Skill provides a structural blueprint for building batch and streaming connectors to external systems like databases, APIs, and queues.

Can I build both batch and streaming connectors using the PySpark DataSource API?▼

Yes, you can build both batch and streaming connectors using the PySpark DataSource API. The Skill guides the implementation of DataSourceStreamReader and DataSourceStreamWriter interfaces alongside batch equivalents to enable end-to-end data transfer workflows.

What is the recommended architecture for implementing Spark data sources in Python?▼

The recommended architecture for Spark data sources uses a flat single-level inheritance structure with in-executor imports. This design minimizes dependencies and ensures reliable connector development across custom protocols and external systems.

Does this PySpark data source approach work for ingesting data from REST APIs and message queues?▼

Yes, this PySpark data source approach works for ingesting data from REST APIs and message queues. The Skill provides real-world patterns for data ingestion and pushing data to external systems for both batch and streaming workloads.

Why do I need in-executor imports when developing PySpark data source connectors?▼

In-executor imports are required when developing PySpark data source connectors to ensure that necessary dependencies are available directly on the executor nodes. This prevents serialization errors and maintains a flat inheritance model during data transfer workflows.