spark-python-data-source

To build a custom PySpark connector, implement the PySpark DataSource API by defining reader and writer classes for batch and streaming modes. This approach handles pagination, partitioning, and type conversion across driver and executor boundaries.

PySpark DataSource supports both batch and streaming ingestion and egress. You implement specific DataSourceReader and DataSourceWriter classes alongside stream variants to handle continuous data flows and exactly-once semantics.

Authentication in a Spark Python data source connector is handled through a multi-method credential fallback. This supports Unity Catalog, default credentials, service principals, API keys, and basic auth across Spark driver and executor nodes.

PySpark connector imports fail on executors when third-party dependencies are not loaded in the correct executor-scoped placement. Enforcing flat single-level inheritance serialization and proper import design ensures your connector works reliably across distributed boundaries.

To ensure exactly-once semantics in a Spark streaming connector, implement a resilient streaming offset design within your stream reader class. This pattern guarantees fault-tolerant data processing and prevents duplicate ingestion during pipeline recoveries.

Error handling in a custom Spark data source is managed through resilient patterns including retries and circuit breakers. Integrating these robust error-handling and testing patterns within your reader and writer implementations ensures reliable external system integration.