What problem does it solve?
This Skill helps you design and improve data pipelines so data is delivered to analytics and ML systems in a dependable, scalable, and validated way.
Core Features & Use Cases
- Pipeline architecture: Choose between batch, streaming, Lambda, and Kappa architectures to match latency and complexity needs, e.g., hourly reporting versus real-time fraud monitoring.
- ETL/ELT workflows: Apply ETL or ELT best practices for incremental processing, auditing, orchestration (e.g., dbt), and transformation documentation.
- Data storage selection: Map workloads to SQL databases, NoSQL models, data lakes, or data warehouses based on consistency, scalability, and analytics requirements.
- Data quality & validation: Define quality dimensions (completeness, accuracy, consistency, timeliness, validity, uniqueness) and implement schema/range/pattern and referential integrity checks, plus profiling and lineage for impact analysis.
Quick Start
Generate a production-ready data pipeline blueprint that uses incremental ELT into a data warehouse with dbt, includes a streaming option with Kafka, and specifies concrete data quality checks and lineage tracking for a customer analytics use case.
