spark-declarative-pipelines

Use CREATE OR REPLACE STREAMING TABLE with FROM STREAM read_files, specifying the cloud storage path, format, and schemaHints. The STREAM keyword is required in streaming table queries, and read_files leverages Auto Loader for incremental file processing.

The modern pyspark.pipelines API (imported as dp) is recommended for new projects and supports Liquid Clustering via cluster_by and explicit Unity Catalog paths. The legacy dlt API uses dlt.read with implicit LIVE schema and dlt.apply_changes instead of dp.create_auto_cdc_flow.

Create a streaming table target, then define a flow with AUTO CDC INTO specifying KEYS, SEQUENCE BY for event ordering, and STORED AS SCD TYPE 2. Query history using the auto-generated __START_AT and __END_AT columns, filtering __END_AT IS NULL for current records.

Liquid Clustering with CLUSTER BY is the recommended approach, replacing PARTITION BY and Z-ORDER with adaptive multi-dimensional clustering. Use PARTITION BY only for regulatory physical separation, partition-level retention drops, or older Delta Lake versions.

Migrate when 80% or more of the logic is SQL-expressible, such as filters, aggregations, joins, and simple CASE expressions. Stay in Python when pipelines rely on complex UDFs, external API calls, custom libraries, or ML model inference.

The error occurs when a streaming table query lacks the STREAM keyword with read_files, producing a batch instead of streaming source. Use FROM STREAM read_files so the source is streaming, especially when feeding downstream AUTO CDC flows.