skills.rest
LaurentPRAT-DB
LaurentPRAT-DBCOMMUNITY
Explore all skills by this author →
GitHub

spark-python-data-source

Develop custom Python data source connectors for Apache Spark 4.0+.

Updated Feb 27, 2026
Search Tags:#streaming#batch processing#connector#spark#pyspark#data source
One-click install
npx skills add https://github.com/LaurentPRAT-DB/LPT_claude_config --skill spark-python-data-source-laurentprat-db
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/LaurentPRAT-DB/LPT_claude_config/tree/main/skills/spark-python-data-source
Command: npx skills add https://github.com/LaurentPRAT-DB/LPT_claude_config --skill spark-python-data-source-laurentprat-db

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill enables the creation of custom data source connectors for Apache Spark, allowing seamless integration with external systems that lack native Spark support.

Core Features & Use Cases

  • Custom Connectors: Develop batch and streaming readers/writers for databases, APIs, and message queues.
  • External System Integration: Connect Spark to systems without native support, implementing specialized protocols or authentication.
  • Use Case: Build a Spark connector to read data from a proprietary REST API that returns data in a custom JSON format, enabling Spark SQL queries on this external data.

Quick Start

Use the spark-python-data-source skill to create a Spark data source for reading from MongoDB with sharding support.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a custom PySpark data source connector for an external API?▼

To build a custom PySpark data source connector, implement the Python DataSource API to define batch and streaming readers or writers for your external API. This enables Spark SQL queries on systems lacking native Spark support.

Can I create a streaming reader and writer for Apache Spark in Python?▼

Yes, you can create a streaming reader and writer for Apache Spark in Python by implementing the PySpark DataSource API. This supports continuous data ingestion and output for message queues and external systems.

Does the PySpark DataSource API support batch processing for external databases?▼

Yes, the PySpark DataSource API supports batch processing for external databases. You can develop custom connectors that handle batch reads and writes, enabling Spark to integrate with proprietary database systems.

What is the best way to connect Spark 4.0 to a proprietary REST API returning custom JSON?▼

The best way to connect Spark 4.0 to a proprietary REST API returning custom JSON is developing a custom Python data source connector. This implements the DataSource API with single-level inheritance for explicit control flow.

Do I need Apache Spark 4.0 to use the Python data source connector API?▼

Yes, you need Apache Spark 4.0 or higher to use this Python data source connector implementation. The Skill specifically targets the PySpark DataSource API introduced in Spark 4.0 for external system integration.

Why use a custom Python data source instead of native Spark connectors?▼

Use a custom Python data source when native Spark connectors are unavailable for your external system. It allows you to implement specialized protocols or authentication for databases, APIs, and message queues.

Related Skills

Explore More Software Engineering Skills→
juanlamadrid20juanlamadrid20

spark-python-data-source

Develop custom Python-based Apache Spark data source connectors for batch and streaming workloads.

Community
Advanced
microsoftmicrosoft
952

spark-consumption-cli

Run interactive PySpark and Spark SQL queries on Fabric Lakehouse Livy sessions.

Official
Advanced
Patasse97Patasse97

senior-data-engineer

Design and orchestrate data pipelines with Python, SQL, Spark, Airflow, and dbt.

Community
Advanced
databrickslabsdatabrickslabs
38

spark-python-data-source

Develops custom Apache Spark 4.0+ Python data source connectors for external systems.

Official
Advanced