skills.rest
mirakui
mirakuiCOMMUNITY
Explore all skills by this author →
Website𝕏@mirakuiGitHub

spark-python-data-source

Build custom Spark Python data sources for batch and streaming external systems.

Updated Sep 9, 2017
Search Tags:#python#streaming#etl#connector#spark#batch-processing#datasource
One-click install
npx skills add https://github.com/mirakui/dotfiles --skill spark-python-data-source-mirakui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/mirakui/dotfiles/tree/main/claude/skills/spark-python-data-source
Command: npx skills add https://github.com/mirakui/dotfiles --skill spark-python-data-source-mirakui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Build custom Spark data sources in Python to read from and write to external systems (databases, APIs, queues) for batch and streaming workloads, enabling seamless integration where native Spark support is lacking.

Core Features & Use Cases

  • Flat, single-level inheritance data source pattern returning readers/writers
  • DataSource entry point with concrete DataSource, DataSourceReader/DataSourceWriter, and DataSourceStreamReader/DataSourceStreamWriter implementations
  • Support for both batch and streaming connectors with straightforward option handling, streaming offsets, and error handling
  • Real-world use cases include external databases, REST APIs, message queues, and custom file formats requiring tailored authentication and type mapping

Quick Start

Implement a Spark data source by creating Python classes that implement DataSource, DataSourceReader, DataSourceWriter, DataSourceStreamReader, and DataSourceStreamWriter following the flat inheritance pattern.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a custom Spark Python data source for an external API?▼

To build a Spark Python data source for an external API, you implement Python classes following a flat, single-level inheritance pattern. This includes creating a DataSource entry point alongside concrete DataSourceReader and DataSourceWriter implementations for batch processing.

When do I need to implement a custom Spark data source in Python?▼

You need a custom Spark data source in Python when connecting to external systems like databases, REST APIs, or custom file formats lacking native Spark support. This approach enables tailored authentication and type mapping for batch and streaming processing.

Does the Spark Python data source pattern support both batch and streaming connectors?▼

Yes, the Spark Python data source pattern supports both batch and streaming connectors through straightforward option handling. It utilizes concrete implementations for readers and writers to manage batch jobs and continuous streaming offsets.

What is the required class structure for a Spark Python data source?▼

The required structure for a Spark Python data source involves a DataSource entry point returning reader and writer classes. It uses flat, single-level inheritance to define concrete DataSource, DataSourceReader, DataSourceWriter, and streaming variants.

Related Skills

Explore More Software Engineering Skills→
databrickslabsdatabrickslabs
38

spark-python-data-source

Develops custom Apache Spark 4.0+ Python data source connectors for external systems.

Official
Advanced
thbehthbeh

spark-python-data-source

Develop custom Apache Spark data source connectors for batch and streaming processing.

Community
Advanced
lamb92009lamb92009

spark-engineer

Build and optimize Apache Spark data pipelines for batch and streaming workloads.

Community
Advanced
RamVegirajuRamVegiraju
4

spark-python-data-source

Build custom Python-based Spark data sources for batch and streaming connectors.

Community
Advanced