data_transform

Configure the model with materialized='incremental' and a unique_key, then wrap a filter in an is_incremental() block that selects only rows newer than the max timestamp in the existing table. This processes only new or changed records on each run.

Use pandas for data under 500MB and prototyping, polars for 500MB-100GB where performance matters (10-100x faster with lazy evaluation), and PySpark for datasets over 100GB requiring distributed cluster processing.

ELT loads raw data into a cloud warehouse first and transforms it with SQL tools like dbt, suiting teams with SQL analysts and changing logic. ETL transforms before loading, which fits compliance requirements like pre-load PII redaction or targets lacking compute power.

Replace read_csv with scan_csv for lazy evaluation, assign with with_columns, string column references with pl.col() expressions, and call collect() to execute the query. The APIs are similar, making migration straightforward for most transformations.

Define generic tests like unique, not_null, accepted_values, and relationships in your model's YAML file, or write singular tests as custom SQL queries in the tests directory. Great Expectations can add further validation such as value range checks.

Choose Airflow for enterprise production with thousands of integrations and managed services like AWS MWAA. Pick Dagster for heavy dbt usage and asset-based lineage, or Prefect for dynamic workflows with runtime task generation.