skills.rest
lucaslessachaves
lucaslessachavesCOMMUNITY
Explore all skills by this author →
GitHub

spark-python-data-source

Build custom PySpark DataSources for batch and streaming Spark I/O.

1|Updated Dec 15, 2025
Search Tags:#streaming#batch#connector#spark#pyspark#datasource#data-source
One-click install
npx skills add https://github.com/lucaslessachaves/default --skill spark-python-data-source-lucaslessachaves
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/lucaslessachaves/default/tree/main/.claude/skills/spark-python-data-source
Command: npx skills add https://github.com/lucaslessachaves/default --skill spark-python-data-source-lucaslessachaves

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, or custom protocol). It covers building a Spark connector or plugin in Python, implementing a DataSourceReader or DataSourceWriter, pulling data from or pushing data to a system via Spark, or working with the PySpark DataSource API in any way. If someone says "read from X in Spark" or "write DataFrame to Y" and there is no native connector, this skill applies.

Core Features & Use Cases

  • Create custom Python data sources for Spark (batch and streaming) using the PySpark DataSource API.
  • Implement readers/writers (DataSourceReader, DataSourceWriter, DataSourceStreamReader/Writer) with proper inheritance and option handling.
  • Integrate Spark with external systems (databases, APIs, queues) to build end-to-end data pipelines and enable incremental processing.

Quick Start

Create a sample PySpark project, implement your custom DataSource following the provided template, and run a local Spark job to validate a batch read

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a custom PySpark data source to read from an external API?▼

Build a custom PySpark data source by implementing the DataSource and DataSourceReader classes from the PySpark DataSource API to handle batch reads. This skill provides templates covering the full lifecycle to connect Spark to external systems like APIs when no native connector exists.

Can I use the PySpark DataSource API for both batch and streaming workloads?▼

Yes, the PySpark DataSource API supports both batch and streaming workloads. You can implement DataSourceReader and DataSourceWriter for batch processing, or use DataSourceStreamReader and DataSourceStreamWriter to handle incremental data pipelines and continuous reads from message queues.

What is the best way to write a Spark DataFrame to a custom protocol in Python?▼

Write a Spark DataFrame to a custom protocol by implementing a custom DataSourceWriter using the PySpark DataSource API. This skill guides you through the flat inheritance pattern and necessary in-executor imports to push data to systems lacking native Spark connectors.

Do I need to manage dependencies differently when building a PySpark connector?▼

Building a PySpark connector requires managing dependencies by enforcing in-executor imports and keeping them minimal. This skill emphasizes robust testing, error handling, and security considerations to ensure your custom data source functions reliably across distributed Spark executors.

When should I not use the PySpark DataSource API for connecting to external systems?▼

You should avoid the PySpark DataSource API when a native Spark connector already exists for your target external system. If a built-in JDBC connector or official plugin handles your database or queue, using those pre-built options is simpler than implementing a custom Python data source from scratch.

Related Skills

Explore More Data & Analytics Skills→
ThomazRossitoThomazRossito
31

spark-python-data-source

Build custom PySpark DataSource implementations for Apache Spark external systems.

Community
Advanced
thbehthbeh

spark-python-data-source

Develop custom Apache Spark data source connectors for batch and streaming processing.

Community
Advanced
Awish021Awish021
10

pyspark-databricks

Build and optimize PySpark ETL pipelines on Databricks with Delta Lake.

Community
Intermediate
slysikslysik

spark-python-data-source

Build custom Spark 4.0+ data source connectors in Python.

Community
Advanced