spark-python-data-source

Implement custom PySpark DataSources for batch and streaming workloads.

Updated Mar 25, 2026

Search Tags:#api #python #streaming #batch #connector #spark #datasource

Ready to run

npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill spark-python-data-source-jingyiwng

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher/tree/main/.claude/skills/spark-python-data-source
Command: npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill spark-python-data-source-jingyiwng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Many external systems lack native Spark connectors, making it hard to read or write data from APIs, databases, or custom protocols in Spark jobs. This skill provides a structured way to implement custom Python DataSources for both batch and streaming workloads.

Core Features & Use Cases

Unified Architecture: Single‑level inheritance DataSource, Reader/Writer, Batch and Stream classes ensure easy serialization across driver and executors.
Authentication Flexibility: Supports Unity Catalog, cloud default credentials, service principals, API keys, and basic auth.
Partitioning Strategies: Time‑based, token‑range, and ID‑range patterns enable parallel reads and writes.
Robust Error Handling: Includes exponential backoff, circuit breaker, dead‑letter queue, and timeout handling.
Testing & Validation: Reference templates, unit test scaffolding, type‑conversion utilities, and production‑grade validation guidelines.

Use case example: Build a connector to read from a paginated REST API and write results to Snowflake, all within a Databricks notebook.

Quick Start

Ask the assistant to create a Spark data source for reading from a MongoDB database with sharding support.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build a custom Spark data source for a REST API that lacks a native connector?▼

To build a custom Spark data source for a REST API, you implement Python DataSource, Reader, and Writer classes using the PySpark DataSource API to handle batch and streaming workloads. This approach enables Spark to read from and write to external systems without native connectors.

Can I use the PySpark DataSource API for both batch and streaming workloads?▼

Yes, the PySpark DataSource API supports both batch and streaming workloads through single-level inheritance DataSource, Reader/Writer, and Batch/Stream classes. This unified architecture ensures easy serialization across Spark driver and executors for custom data integration.

What partitioning strategies are available when creating a Python Spark data source?▼

When creating a Python Spark data source, you can implement time-based, token-range, and ID-range partitioning strategies. These patterns enable parallel reads and writes to optimize data ingestion from external APIs, databases, or custom protocols into Spark.

How do I handle authentication in a custom PySpark data source connector?▼

You handle authentication in a custom PySpark data source connector by supporting Unity Catalog, cloud default credentials, service principals, API keys, and basic auth. This flexibility ensures secure access when reading from and writing to external systems.

What is the best way to manage error handling and retries in Spark data source connectors?▼

The best way to manage error handling in Spark data source connectors is to implement exponential backoff, circuit breakers, dead-letter queues, and timeout handling. These robust error handling mechanisms ensure reliable streaming and batch processing.

Do I need to implement custom classes for reading and writing separately in PySpark?▼

Yes, you need to implement separate Reader and Writer classes with flat inheritance for your PySpark data source. This structure, alongside Batch and Stream classes, allows Spark to correctly interface with external systems for reading and writing operations.