expectations

Enforce data quality expectations in pipelines via YAML definitions.

1|1|Updated Mar 4, 2026

Search Tags:#validation #yaml #macros #pipelines #data-quality #expectations

Ready to run

npx skills add https://github.com/starlake-ai/starlake-skills --skill expectations-starlake-ai

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: expectations
Source: https://github.com/starlake-ai/starlake-skills/tree/main/skills/expectations
Command: npx skills add https://github.com/starlake-ai/starlake-skills --skill expectations-starlake-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill enables teams to define and enforce data quality checks across data pipelines, ensuring data correctness and reliability.

Core Features & Use Cases

Built-in macros for common quality checks (e.g., is_col_value_not_unique, is_row_count_to_be_between, count_by_value)
YAML-based definitions via table.sl.yml (load) or task.sl.yml (transform) to express expectations that trigger warnings or errors
Custom macros and extensibility through metadata/expectations to tailor checks to domain-specific rules

Quick Start

Create a simple set of expectations in table.sl.yml to enforce a primary-key uniqueness constraint.

Frequently Asked Questions about expectations

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I enforce data quality checks across data pipelines using YAML?▼

You can enforce data quality checks by defining expectations in YAML files like table.sl.yml or task.sl.yml. These definitions validate schemas, row counts, and value distributions during loading and transformation, triggering warnings or errors when checks fail.

What built-in macros are available for validating schema and row counts?▼

Built-in macros for schema and row count validation include is_col_value_not_unique, is_row_count_to_be_between, and count_by_value. These built-in functions allow you to enforce primary-key uniqueness and verify data distribution constraints directly in your pipelines.

Can I use custom macros for domain-specific data validation rules?▼

Yes, you can extend data validation with custom macros stored in the metadata/expectations directory. This allows teams to tailor quality checks to specific domain rules beyond the standard built-in macro library.

When do I need to apply data quality expectations during loading versus transformation?▼

Apply data quality expectations during loading using table.sl.yml to validate raw inputs, and during transformation using task.sl.yml to validate processed outputs. Both stages enforce schema correctness and value distributions to ensure pipeline reliability.

Does defining data quality expectations require external pipeline dependencies?▼

No, defining data quality expectations does not require external dependencies. The validation framework operates natively using YAML-based definitions and macros to enforce checks without integrating third-party data quality tools.

What is the best way to trigger an error when a primary-key uniqueness constraint fails?▼

To trigger an error for primary-key uniqueness, define an expectation in table.sl.yml using the is_col_value_not_unique macro. This built-in check validates uniqueness during data loading and automatically raises an error if duplicates are detected.