data-validation

Data validation across ETL pipelines is enforced by applying schema validation, type checks, and business rules at ingestion, transformation, and output boundaries. This prevents bad data from propagating by using Pydantic models and SQL constraints.

Data contracts are defined by codifying schema expectations and business rules using Pydantic models and Great Expectations patterns. When data fails these checks, dead-letter records are surfaced and alerts are triggered to catch bad records.

API inputs are validated using Pydantic-based schema validation for Python data models. This approach applies strict type checks and structural constraints to ensure incoming API payloads meet defined data quality boundaries.

Great Expectations-inspired checks can be applied for batch validation across data processing jobs. The validation strategy performs checks on datasets, surfacing failures and dead-letter records when data does not meet the defined constraints.

SQL constraints prevent bad data propagation by enforcing strict schema and type rules directly within the pipeline transformation and output stages. Combining them with Python models ensures end-to-end data quality across all boundaries.

Data validation contracts are needed because transformations alone do not guarantee data quality. Explicitly defining and enforcing contracts at every boundary detects and prevents bad data, ensuring consistent schema validation and type checks across the pipeline.