metodologia-data-engineering

What problem does it solve?

Data engineering teams often struggle to design, communicate, and implement robust, end-to-end data platforms that span ingestion, orchestration, storage, quality, lineage, and cost management. This Skill provides a structured framework and reference patterns to architect data platforms that are scalable, auditable, and vendor-agnostic.

Core Features & Use Cases

• Ingestion patterns and data contracts: defines ingestion strategies (batch, CDC, streaming) and formal data contracts to ensure data quality and clear ownership.
• Orchestration and storage architecture: guides orchestration choices (Dagster, Airflow) and a lakehouse storage design (landing/curated/marts) with open formats like Iceberg or Delta.
• Data quality, lineage, observability, and cost management: prescribes a unified approach to validation, lineage capture, monitoring, alerting, and cost attribution.
• Broad coverage including edge cases: supports greenfield projects, legacy migrations, multi-cloud setups, real-time streaming, and compliance considerations.

Quick Start

Provide the system or project name as input to generate the end-to-end data platform architecture.