data-quality

Data quality validation in ETL pipelines identifies completeness, accuracy, and consistency issues before data reaches analytics or ML systems. This Skill automates checks using Great Expectations to detect schema violations, null values, and cross-field anomalies, then triggers alerts on failure.

Yes. This Skill supports both batch and streaming data workflows, applying automated validation rules to detect quality issues in real time or post-ingestion. It integrates directly into pipelines and alerts when thresholds are breached.

You can define rules for completeness (no nulls), accuracy (value ranges), consistency (cross-field logic), timeliness, and schema validation. For example, validate that user_id is not null and age falls within 0–120, catching anomalies before downstream use.

This Skill validates data during lake-to-warehouse integrations, applying SLA-based monitoring and automated rule enforcement. It catches quality issues at ingestion, enabling early remediation and preventing bad data from reaching reporting or ML systems.

No. This Skill uses Great Expectations, a widely adopted framework that supports automated rule definition without writing validation code from scratch. You configure expectations declaratively and integrate them into your pipeline workflow.

When validation thresholds are breached, the Skill triggers alerts and generates dashboards for monitoring. You can configure actions to pause downstream processing, log failures, or notify teams, enabling rapid detection and remediation of data quality issues.