senior-data-engineer

Define the source schema, generate an Airflow DAG for extraction, create dbt staging and incremental mart models with a unique_key and is_incremental filter, then add schema tests for uniqueness, not-null, and accepted ranges. Schedule the DAG with retries and alerting.

Choose streaming when you need second-to-minute latency on continuous event data, and batch for hourly or daily transformations of large historical datasets. Batch is more cost-effective; streaming requires careful watermarking and exactly-once design.

Lambda maintains separate batch and speed layers with merged serving views, suiting ML training plus real-time serving. Kappa uses a single stream-processing path with replay from an immutable Kafka log, reducing code duplication when all logic can be expressed as streams.

Use an incremental model with the check strategy and check_cols listing tracked attributes, generating surrogate keys and effective start/end dates. Type 2 inserts a new row per change while closing the prior record; Type 1 simply overwrites in place.

OOM errors usually come from undersized executor memory or skewed partitions. Increase spark.executor.memory, raise spark.sql.shuffle.partitions to spread data, enable Kryo serialization, and repartition skewed join keys before wide transformations.

Add a watermark with withWatermark on the event-time column, such as 10 minutes, so state is dropped after the threshold. Records arriving later than the watermark are discarded; route them to a dead letter queue via foreachBatch if they must be captured.