spark-python-data-source

Implement a DataSource subclass with name, schema, reader, and writer methods, then register it with spark.dataSource.register. Readers extend DataSourceReader or DataSourceStreamReader, and writers extend DataSourceWriter or DataSourceStreamWriter, using a flat single-level inheritance structure.

Extend DataSourceStreamReader and implement initialOffset, latestOffset, partitions(start, end), and commit. Offsets must be JSON-serializable, and partitions should be non-overlapping—typically by adjusting boundaries by one microsecond to prevent duplicate records between batches.

Use time-based partitions for temporal APIs, token-range partitioning for distributed databases like Cassandra, and ID-range partitioning for paginated APIs. Aim for 128MB to 1GB per partition and 2-4x executor cores for batch reads.

Yes, through idempotent writes in DataSourceStreamWriter. Generate deterministic idempotency keys from batch ID, partition ID, and row content, then skip records already written, combined with commit and abort callbacks for batch lifecycle management.

Mock HTTP calls with unittest.mock.patch for unit tests against readers and writers. For integration tests, use Testcontainers to spin up real systems like PostgreSQL and verify end-to-end reads and writes with a local Spark session.

Support multiple methods in priority order: Databricks Unity Catalog credentials, cloud default credentials like managed identity, service principals, API keys, and username/password. Validate that at least one method is configured and never log sensitive values.