skills.rest
RamVegiraju
RamVegirajuCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

spark-python-data-source

Build custom Python-based Spark data sources for batch and streaming connectors.

4|4|Updated Jan 5, 2026
Search Tags:#streaming#batch#connector#spark#pyspark#datasource
One-click install
npx skills add https://github.com/RamVegiraju/databricks-samples --skill spark-python-data-source-ramvegiraju
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/RamVegiraju/databricks-samples/tree/main/.claude/skills/spark-python-data-source
Command: npx skills add https://github.com/RamVegiraju/databricks-samples --skill spark-python-data-source-ramvegiraju

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Build custom Python-based Spark data sources to read from and write to external systems, enabling seamless integration with batch and streaming workloads.

Core Features & Use Cases

  • Implements the PySpark DataSource API with flat inheritance: DataSource, DataSourceReader/Writer, and DataSourceStreamReader/Writer.
  • Supports batch and streaming modes for connectors to external databases, APIs, or file formats.
  • Provides guidance on partitioning, authentication, type conversion, and offset management for reliable data ingestion.

Quick Start

Implement the DataSource, Readers, and Writers following the pattern, then register the new format with Spark to use it for batch and streaming reads and writes.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a custom PySpark data source to read from an external API?▼

You can build a custom PySpark data source by implementing the DataSource, DataSourceStreamReader, and DataSourceStreamWriter classes. This pattern supports streaming ingestion from external systems by managing offsets and partitioning for reliable data processing.

What is the best way to build a Spark connector for batch and streaming workloads in Python?▼

The PySpark DataSource API requires a flat inheritance pattern implementing DataSource, DataSourceReader, and DataSourceWriter. You must use explicit imports and standard-library-first dependencies to ensure reliable integration with external systems.

Do I need external dependencies to build a custom Spark data source with Python?▼

The custom Spark data source implementation handles type conversion and authentication natively. By extending the dedicated reader and writer classes, you can configure these mechanisms to securely connect to external databases and APIs.

Can I use the PySpark DataSource API for both batch and streaming scenarios?▼

The PySpark DataSource API supports both batch and streaming scenarios through dedicated classes. It uses a flat inheritance pattern with DataSource, DataSourceReader, and DataSourceWriter to connect Spark workloads to external sinks.

Related Skills

Explore More Data & Analytics Skills→
ThomazRossitoThomazRossito
31

spark-python-data-source

Build custom PySpark DataSource implementations for Apache Spark external systems.

Community
Advanced
m19cm19c

spark-python-data-source

Create custom Apache Spark data sources with PySpark for batch and streaming operations.

Community
Advanced
aehrcaehrc
133

Apache Spark Catalyst API

Create custom expressions and optimize queries in Apache Spark Catalyst.

Official
Advanced
juanlamadrid20juanlamadrid20

spark-python-data-source

Develop custom Python-based Apache Spark data source connectors for batch and streaming workloads.

Community
Advanced