skills.rest
thbeh
thbehCOMMUNITY
Explore all skills by this author →
GitHub

spark-python-data-source

Develop custom Apache Spark data source connectors for batch and streaming processing.

Updated May 31, 2026
Search Tags:#Spark#batch processing#data source#streaming processing#external systems integration
One-click install
npx skills add https://github.com/thbeh/coding-agents-databricks-apps --skill spark-python-data-source-thbeh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/thbeh/coding-agents-databricks-apps/tree/main/.claude/skills/spark-python-data-source
Command: npx skills add https://github.com/thbeh/coding-agents-databricks-apps --skill spark-python-data-source-thbeh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyspark, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables the creation of custom Spark data source connectors, allowing users to integrate external systems (databases, APIs, message queues) with Apache Spark, supporting batch and streaming data processing.

Core Features & Use Cases

  • Custom Data Source Creation: Implement custom data source connectors for databases, APIs, and message queues.
  • Batch & Streaming Processing: Enable batch and streaming data processing capabilities.
  • Use Case: For instance, a user might want to connect Spark to a MongoDB database for batch data processing or to a Kafka message queue for real-time streaming data processing.

Quick Start

Use the spark-python-data-source skill to create a data source for MongoDB and perform a batch read operation.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a custom Spark data source connector for an external system?▼

To build a custom Spark data source connector, use PySpark to implement data source interfaces for external systems. This enables integrating databases, APIs, and message queues for both batch and streaming data processing within Apache Spark.

Can I use PySpark to connect Spark to a MongoDB database for batch processing?▼

Yes, you can use PySpark to connect Spark to a MongoDB database for batch processing. You achieve this by developing a custom data source connector that handles the batch read operations from the external database.

Does Spark support streaming data processing from external message queues like Kafka?▼

Spark supports streaming data processing from external message queues like Kafka. You build a custom data source connector using PySpark to enable real-time streaming data ingestion from the message queue.

What is the best way to integrate external APIs with Apache Spark?▼

The best way to integrate external APIs with Apache Spark is creating a custom data source connector. This approach handles data ingestion from the APIs, supporting both batch and streaming processing workflows.

Do I need PySpark to create a custom data source connector for Spark?▼

Yes, you need PySpark to create a custom data source connector for Spark. It provides the required Python libraries and interfaces for building and configuring data sources for external systems integration.

Related Skills

Explore More Software Engineering Skills→
clawdataclawdata
24

spark

Automate building and running Apache Spark batch and streaming data processing jobs.

Official
Intermediate
thbehthbeh

databricks-spark-declarative-pipelines

Automate Spark data pipeline development and execution on Databricks.

Community
Advanced
Patasse97Patasse97

senior-data-engineer

Design and orchestrate data pipelines with Python, SQL, Spark, Airflow, and dbt.

Community
Advanced
databrickslabsdatabrickslabs
38

spark-python-data-source

Develops custom Apache Spark 4.0+ Python data source connectors for external systems.

Official
Advanced