databricks-spark-declarative-pipelines

To build a Databricks Lakehouse pipeline, you can use this skill to standardize medallion bronze/silver/gold architectures. It enables end-to-end data pipelines across environments by generating asset bundles with serverless compute.

Migrating from Delta Live Tables to Spark Declarative Pipelines is supported with built-in guidance, best practices, and tooling. This skill ensures smooth transitions across environments while enforcing language selection and safe defaults.

Setting up Change Data Capture and SCD Type 2 in Databricks SDP involves implementing streaming patterns and AUTO CDC. This skill provides the configuration to maintain robust lineage and historical data tracking.

Yes, you can use Auto Loader for data ingestion in Spark Declarative Pipelines. The skill supports ingesting data from Auto Loader and other cloud storage sources to initialize your SDP project.

Serverless compute is required to run Databricks Lakeflow SDP pipelines as configured by this skill. It streamlines the creation and maintenance of scalable data pipelines with minimal boilerplate.

Limitations when configuring streaming workflows with Databricks SDP relate to enforcing safe defaults and language selection between SQL or Python. The skill applies modern SDP best practices to manage these constraints across environments.