skills.rest
datasciencemonkey
datasciencemonkeyCOMMUNITY
Explore all skills by this author →
GitHub

spark-python-data-source

Build custom Python Spark data sources for external systems.

27|9|Updated Feb 3, 2026
Search Tags:#python#streaming#spark#pyspark#datasource#connectors
One-click install
npx skills add https://github.com/datasciencemonkey/coding-agents-databricks-apps --skill spark-python-data-source
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/datasciencemonkey/coding-agents-databricks-apps/tree/main/.claude/skills/spark-python-data-source
Command: npx skills add https://github.com/datasciencemonkey/coding-agents-databricks-apps --skill spark-python-data-source

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Build custom Spark data sources for external systems, enabling Spark to read from and write to databases, APIs, and message queues that lack native connectors.

Core Features & Use Cases

  • Flat Python-based Spark DataSource pattern with explicit DataSource, DataSourceReader, DataSourceWriter, and their streaming variants.
  • Supports batch and streaming connectors to external systems, with minimal dependencies and clear, readable implementation patterns.
  • Use case: Create a connector to a REST API or a database, enabling Spark to ingest live data and push updates.

Quick Start

Create a Python-based Spark data source following the provided skeleton to connect Spark to an external system.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a custom Spark data source in Python for an external system?▼

To create a custom Spark data source in Python, implement a flat inheritance pattern using DataSource, DataSourceReader, and DataSourceWriter classes to connect Spark to external systems lacking native support.

Can I build streaming connectors for Spark using Python?▼

Yes, you can build streaming connectors for Spark using Python by implementing the streaming variants of DataSourceReader and DataSourceWriter to ingest live data from message queues and APIs.

When do I need a custom PySpark data source instead of native connectors?▼

You need a custom PySpark data source when connecting to databases, REST APIs, or message queues that lack native Spark support, enabling both batch and streaming data ingestion.

What is the best way to structure a Python Spark data source implementation?▼

The best way to structure a Python Spark data source is using a flat inheritance pattern with single-responsibility design, enforcing explicit imports for DataSource, DataSourceReader, and DataSourceWriter components.

Does this Python Spark data source pattern support both batch and streaming workloads?▼

Yes, this Python Spark data source pattern supports both batch and streaming workloads by providing standard reader and writer classes alongside their dedicated streaming variants for external system connectivity.

How do I write Spark data to an external API using a custom Python connector?▼

You write Spark data to an external API by implementing the DataSourceWriter class within your custom Python data source, following the flat inheritance pattern to push updates to systems without native Spark support.

Related Skills

Explore More Data & Analytics Skills→
raphaelchristiraphaelchristi
43

harness:setup

Configure Harness Evolver, LangSmith, and baseline evaluations for Python projects.

Community
Advanced
RamVegirajuRamVegiraju
4

spark-python-data-source

Build custom Python-based Spark data sources for batch and streaming connectors.

Community
Advanced
EstomEstom

spark-engineer

Optimize Apache Spark applications with PySpark or Scala code snippets.

Community
Advanced
JingyiWngJingyiWng

spark-python-data-source

Implement custom PySpark DataSources for batch and streaming workloads.

Community
Advanced