What problem does it solve? Data pipelines often ingest incomplete, duplicated, or invalid records that silently corrupt downstream Silver and Gold tables. This Skill provides ready-to-use validation patterns for PySpark/Delta pipelines so bad data is caught, dropped, or blocked before it propagates. ## Core Features & Use Cases - PySpark Validation Reports: Run structural checks on DataFrames covering required columns, null percentages, and duplicate counts, returning a pass/fail report. - Lakeflow/SDP Expectations: Apply declarative data quality constraints using the modern pyspark.pipelines API (dp.expect, dp.expect_or_drop, dp.expect_all) or native SQL CONSTRAINT ... EXPECT clauses with ON VIOLATION policies. - Source-to-Target Reconciliation: Compare row counts and aggregate sums between staging and final tables after a load to detect data loss. - Use Case: After ingesting bronze sales data, define a Silver streaming table with expectations that drop rows missing event dates and fail the update on null IDs, then reconcile totals against the source. ## Quick Start Ask the agent to add data quality expectations and null checks to my Silver layer pipeline for the bronze_vendas table.
