data-eng

Medallion Architecture organizes data pipelines into Bronze, Silver, and Gold layers. It transforms raw ingested data into cleansed, conformed assets, then aggregates them into business-ready metrics using open table formats like Delta Lake or Iceberg.

To build reliable ETL pipelines with enforced data quality, you can use frameworks like Great Expectations alongside dbt-core. This combination enables data contract enforcement, SLA monitoring, and automated lineage tracking across your transformations.

Processing streaming data with Kafka involves building event-driven pipelines using the kafka-python client. This allows you to ingest real-time events and process them through stream processing frameworks into your lakehouse storage.

Yes, PySpark and dbt-core work together for ELT transformations. PySpark handles distributed data processing, while dbt-core manages the SQL-based data modeling and transformation logic within your lakehouse architecture.

The best way to ingest raw JSON into a bronze layer is to build an automated data pipeline that reads from your source path and writes to cloud storage like S3, ensuring the pipeline is idempotent and observable for reliable raw data landing.

Great Expectations enforces data contracts in ETL pipelines by defining explicit data quality rules and validation checks. This ensures raw data conforms to expected schemas and standards before moving to downstream Silver or Gold layers.