data-engineering

To build a daily ETL process, you extract data from a transactional database, load it into a data warehouse like Snowflake or BigQuery, and transform it using dbt. You can orchestrate this daily ETL workflow using an Airflow DAG.

Designing data pipelines for both batch and streaming processing involves using tools like Airflow for batch orchestration and Kafka for streaming data ingestion. This approach supports robust data infrastructure for continuous and scheduled data flows.

dbt fits into the modern data stack by handling the transformation layer of data modeling. It applies dimensional modeling and star schema concepts within data warehouses like Snowflake to structure data for business intelligence tools.

Yes, you can use Airflow with Snowflake and BigQuery for data pipeline orchestration. Airflow schedules and manages the workflows, while Snowflake and BigQuery serve as the data warehousing storage and computing layers.

You ensure data quality and optimize queries in a data warehouse by applying validation rules, monitoring quality metrics, and using partitioning and query optimization techniques. This maintains robust data infrastructure performance.

The modern data stack consists of ingestion tools like Fivetran, storage layers like Snowflake, transformation tools like dbt, and serving layers like Looker. These components integrate to build and manage comprehensive data pipelines.