skills.rest
databrickslabs
databrickslabsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

spark-python-data-source

Develops custom Apache Spark 4.0+ Python data source connectors for external systems.

38|12|Updated Apr 24, 2026
Search Tags:#streaming#batch processing#connector#spark#pyspark#databricks#data source
One-click install
npx skills add https://github.com/databrickslabs/coding-agents-databricks-apps --skill spark-python-data-source-databrickslabs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-python-data-source
Source: https://github.com/databrickslabs/coding-agents-databricks-apps/tree/main/.claude/skills/spark-python-data-source
Command: npx skills add https://github.com/databrickslabs/coding-agents-databricks-apps --skill spark-python-data-source-databrickslabs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill eliminates the difficulty of connecting Apache Spark to external systems (databases, APIs, message queues, custom file formats) that lack native Spark support, removing the need to build error-prone, unmaintainable custom integration code from scratch.

Core Features & Use Cases

  • Batch & Streaming Support: Build connectors that handle both historical batch data processing and real-time streaming data ingestion for continuous analytics use cases.
  • Flexible Parallel Partitioning: Implement optimized parallel reads using time-based, token-range, or ID-range partitioning strategies to maximize Spark cluster utilization.
  • Production-Grade Guardrails: Includes built-in guidance for multi-method authentication, error resilience, type conversion, and observability to deploy connectors in production Databricks environments. For example, use this Skill to build a streaming connector that reads from a RabbitMQ message queue with at-least-once delivery guarantees and writes data to a Delta table for real-time monitoring.

Quick Start

Use the spark-python-data-source skill to build a custom Spark connector that reads paginated data from your internal REST API with OAuth2 authentication and writes it to a Delta table for analytics.

Frequently Asked Questions about spark-python-data-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a custom Spark data source connector for an external system without native support?▼

Build custom Spark data source connectors using Python to handle external systems lacking native Spark support. This covers batch and streaming read/write operations with parallel partitioning, multi-method authentication, and bidirectional type conversion for Databricks integration.

Can I implement streaming read and write operations with exactly-once semantics in a PySpark connector?▼

Implement streaming read and write operations with exactly-once semantics in a PySpark connector. The Skill provides production-grade guidance for continuous analytics ingestion and output, including error resilience and observability for Databricks environments.

Does Apache Spark 4.0 support custom Python data source connectors for batch processing?▼

Apache Spark 4.0 supports custom Python data source connectors for batch processing. You can implement optimized parallel reads using time-based, token-range, or ID-range partitioning strategies to maximize cluster utilization for historical data.

What is the best way to connect a Spark streaming job to a RabbitMQ message queue?▼

Connect Spark streaming to external message queues by building a custom Python data source connector. This approach enables reading from systems like RabbitMQ with at-least-once delivery guarantees and writing data to Delta tables for real-time monitoring.

How do I handle OAuth2 authentication when reading paginated data from a REST API in Spark?▼

Handle OAuth2 authentication and read paginated data from REST APIs by developing a custom Spark Python data source connector. The Skill includes built-in guidance for multi-method authentication and bidirectional type conversion between Spark and external formats.

What are the limitations of using custom Python connectors for Spark data sources?▼

Custom Python connectors for Spark data sources require careful implementation of error resilience and observability to meet production requirements. You must manage bidirectional type conversion and partitioning strategies manually to avoid unmaintainable, error-prone integration code.

Related Skills

Explore More Software Engineering Skills→
neo4j-contribneo4j-contrib
101

neo4j-spark-skill

Read and write Neo4j data with Apache Spark DataFrames.

Official
Advanced
yunqiqiliangyunqiqiliang
8

clickzetta-realtime-sync-pipeline

Automates real-time single-table data synchronization from Kafka, MySQL, PostgreSQL, or SQL Server into Lakehouse.

Community
Advanced
Zack2626-okZack2626-ok

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Spark Declarative Pipelines.

Community
Advanced
juanlamadrid20juanlamadrid20

spark-python-data-source

Develop custom Python-based Apache Spark data source connectors for batch and streaming workloads.

Community
Advanced