skills.rest
slysik
slysikCOMMUNITY
Explore all skills by this author →
GitHub

spark-python-data-source

Build custom Spark 4.0+ data source connectors in Python.

Updated Mar 10, 2026
Search Tags:#streaming#batch processing#connector#spark#pyspark#data source
One-click install
npx skills add https://github.com/slysik/databricks-claude-coding --skill spark-python-data-source-slysik
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/slysik/databricks-claude-coding/tree/main/.agents/skills/spark-python-data-source
Command: npx skills add https://github.com/slysik/databricks-claude-coding --skill spark-python-data-source-slysik

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill enables the creation of custom Apache Spark data source connectors for external systems, bridging the gap where native Spark support is missing.

Core Features & Use Cases

  • Custom Connectors: Build readers and writers for databases, APIs, message queues, or custom file formats.
  • Batch & Streaming: Implement both batch and streaming data processing capabilities.
  • Use Case: Develop a Spark connector to read data from a proprietary API that doesn't have native Spark integration, allowing seamless data analysis within Spark.

Quick Start

Use the spark-python-data-source skill to build a custom Spark connector for reading data from a REST API.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a custom Spark data source connector for a REST API?▼

Yes, you can use this Skill to implement custom Spark streaming readers and writers for message queues. It supports building both batch and streaming connectors in Python for external systems without native Spark support.

When do I need a custom PySpark connector for external systems?▼

This Skill requires Spark 4.0+ and Python. It is designed for environments where external systems lack native Spark integration, utilizing a flat, single-level inheritance pattern for simple, explicit connector implementations.

Does this approach to building Spark connectors use complex abstractions?▼

No, this approach emphasizes simple, explicit implementations over complex abstractions. It adheres to a flat, single-level inheritance pattern to facilitate building custom Spark data source connectors cleanly in Python.

What's the best way to read data from a proprietary API into Spark?▼

The best way to read data from a proprietary API into Spark is by developing a custom data source connector. This Skill enables you to build Python readers and writers for seamless data analysis within your Spark environment.

Related Skills

Explore More Software Engineering Skills→
alessandro9110alessandro9110
4

spark-python-data-source

Develop custom Apache Spark 4.0+ data source connectors in Python.

Community
Advanced
thbehthbeh

databricks-spark-declarative-pipelines

Automate Spark data pipeline development and execution on Databricks.

Community
Advanced
clawdataclawdata
24

spark

Automate building and running Apache Spark batch and streaming data processing jobs.

Official
Intermediate
databrickslabsdatabrickslabs
38

spark-python-data-source

Develops custom Apache Spark 4.0+ Python data source connectors for external systems.

Official
Advanced