software-engineering-eng-data

What problem does it solve?

This Skill helps you design reliable data engineering pipelines and data infrastructure decisions by mapping work to the data lifecycle while enforcing schema contracts, data quality, and observability.

Core Features & Use Cases

• Data Lifecycle Pipeline Design: Guides end-to-end pipeline planning across generation, storage, ingestion (batch/streaming), transformation (ELT), and serving with the right technical approach for each stage.
• Data Quality & Testing Frameworks: Establishes data quality dimensions (accuracy, completeness, consistency, timeliness, uniqueness, validity) and translates them into concrete checks and operational thresholds.
• Schema Evolution & Data Contracts: Provides safe, backward-compatible evolution protocols and contract expectations (schema, semantics, SLOs, ownership) to prevent breaking downstream consumers.
• Batch vs Streaming Decisions: Uses a decision matrix and streaming semantics (idempotency/exactly-once strategy) to choose the right architecture.

Quick Start

Ask it: "Design a production-ready data pipeline from our PostgreSQL source to our warehouse, including batch vs streaming choice, schema evolution plan, data quality tests, and observability SLOs."