skills.rest
omeraltn
omeraltnCOMMUNITY
Explore all skills by this author →
GitHub

agency-data-engineer

Build ETL/ELT pipelines with bronze, silver, and gold lakehouse layers.

Updated Apr 11, 2026
Search Tags:#streaming#dbt#etl#data-quality#data-engineering#lakehouse#delta-lake
One-click install
npx skills add https://github.com/omeraltn/ice_cream_website_testing --skill agency-data-engineer-omeraltn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agency-data-engineer
Source: https://github.com/omeraltn/ice_cream_website_testing/tree/main/.antigravity/agency-data-engineer
Command: npx skills add https://github.com/omeraltn/ice_cream_website_testing --skill agency-data-engineer-omeraltn

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

The Agency Data Engineer skill solves the challenge of reliably turning raw, heterogeneous source data into observable, analytics-ready tables so BI and ML teams can trust and act on data without manual intervention or silent failures.

Core Features & Use Cases

  • Robust ETL/ELT Pipelines: Idempotent ingestion, CDC/incremental processing, and bronze→silver→gold layering for clear data contracts.
  • Data Quality & Observability: Integrated schema contracts, automated validations, Great Expectations checks, lineage, and SLA-based alerting to detect drift and failures quickly.
  • Streaming & Cloud Integration: Patterns and examples for Kafka/Spark streaming, Delta/Iceberg table strategies, partitioning and compaction optimizations, and cloud-native lakehouse deployments.
  • Use Case: Ingest JSON order events from Kafka into a bronze append-only store, apply deduplication and conformance in silver, and publish daily revenue aggregates to gold with monitoring and SLA enforcement.

Quick Start

Create a bronze→silver→gold pipeline that ingests source events, enforces schema contracts and data quality checks, and outputs a daily aggregated revenue table with freshness SLAs.

Frequently Asked Questions about agency-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an ETL pipeline with bronze, silver, and gold layers?▼

To build a bronze, silver, and gold ETL pipeline, ingest raw source data into an append-only bronze store, apply deduplication and conformance in silver, and publish aggregated analytics tables to gold with freshness SLAs and monitoring.

How does CDC and incremental processing work in a lakehouse architecture?▼

CDC and incremental processing in a lakehouse capture source changes continuously, applying idempotent ingestion and schema evolution handling to update Delta or Iceberg tables without reprocessing the entire dataset.

Can I use dbt with Delta Lake for automated data quality validation?▼

Yes, dbt can define explicit schema contracts and transformations on Delta Lake tables, while integrating automated data quality checks and Great Expectations validations to detect drift and enforce SLAs.

What is the best way to handle schema evolution in streaming ETL pipelines?▼

Handling schema evolution in streaming ETL pipelines requires explicit schema contracts and automated validation during the silver layer transformation to ensure heterogeneous source data conforms before publishing to gold.

Does this approach support Kafka and Spark streaming for real-time analytics?▼

Yes, this approach supports Kafka and Spark streaming to ingest JSON events continuously into a lakehouse, applying CDC, partitioning, and compaction optimizations for analytics-ready tables.

How do I enforce data observability and lineage in batch and streaming pipelines?▼

Data observability and lineage are enforced through integrated schema contracts, automated validations, Great Expectations checks, and SLA-based alerting to detect drift and failures across batch and streaming scenarios quickly.

Related Skills

Explore More Data & Analytics Skills→
DKSangDKSang
2

des-bronze-layer-design

Designs a Bronze layer specification for raw storage, metadata, retention, and schema drift.

Community
Advanced
jgtolentinojgtolentino

insightpulse-deepnote-data-lab

Design and operate Deepnote data lab workspaces for Supabase/Postgres analytics.

Community
Advanced
BoraPerusicBoraPerusic

data-quality-frameworks

Validate data pipelines with Great Expectations, dbt tests, and data contracts.

Community
Advanced
elophantoelophanto
116

data-engineering

Build end-to-end data pipelines transforming raw data into analytics-ready assets.

Community
Advanced