validate-ingestion

To validate Bronze ingestion configs against LLD specifications, run a static analysis check that cross-references per-table YAML configurations with LLD task lists. This verifies config keys, schema definitions, and referenced contract files to ensure full compliance before deployment.

Static analysis for data pipeline ingestion code is an automated validation process that scans runner, factory, and wrapper scripts for syntax errors, hardcoded credentials, and absolute paths. It prevents broken data pipelines by catching critical issues before code is merged into production.

Config validation does check Unity Catalog table naming conventions. It enforces required metadata column definitions and verifies that output table names match expected schemas, reporting any naming convention violations as warnings or critical findings in the validation report.

You can validate Spark Expectations import wrappers in your ingestion framework. The validation process scans for proper fail-closed import patterns, confirms required runtime dependencies are present, and checks that test modules exist to ensure reliable data quality rule execution.

You do need pyyaml installed to run LLD validation checks. The dependency is required to parse and cross-check the per-table YAML configuration files against the low-level design specifications during the static analysis process.

Your Bronze ingestion pipeline may fail on missing contract or DQ rule files because the framework requires these artifacts to match expected schemas. Validation checks confirm file existence and schema compliance, reporting missing files as critical issues to address before merging.