Design batch, streaming, and hybrid data pipelines with quality gates and recovery paths.
npx skills add https://github.com/simota/agent-skills --skill stream-simota
Please help me install this Agent Skill. Skill: Stream Source: https://github.com/simota/agent-skills/tree/main/stream Command: npx skills add https://github.com/simota/agent-skills --skill stream-simota
This Skill addresses the complexity of designing, building, and maintaining robust data pipelines, ensuring data quality, reliability, and efficient flow across various systems.
Use the Stream skill to design a hybrid data pipeline for processing user clickstream data with sub-minute latency requirements.
High-intent search queries and answers about installing and using this skill.
Designing a hybrid data pipeline combines batch processing for bulk loads with streaming architectures like Kafka for real-time ingestion. This approach uses explicit quality gates and schema evolution to manage data flow efficiently across both modes.
Ensuring data quality in an ETL pipeline requires implementing explicit quality gates and idempotency checks during data flow. Defining clear recovery paths and schema evolution strategies guarantees reliable processing and prevents duplicate records.
You can use dbt for data modeling and Airflow for workflow orchestration within an ELT pipeline. This combination allows you to schedule transformations, visualize data lineage, and enforce quality management across your data systems.
Schema evolution and recovery paths are needed when your data pipeline handles changing data structures or requires high reliability. They ensure your ETL or streaming processes adapt to schema modifications and recover gracefully from failures.
Choosing between batch processing and streaming for clickstream data depends on your latency requirements. If you need sub-minute insights, a streaming pipeline using Kafka is optimal, whereas batch processing suits scheduled, less time-sensitive analytics workloads.
JaviMontanoDesign a data quality framework for profiling, validation, anomaly detection, contracts, and SLA monitoring.
CODE-SAURABHDesign reliable data pipelines with idempotent processing and schema validation.
SSiertsemaDefine data governance policies covering stewardship, quality, access, lifecycle, lineage, and metadata.
chicanoandres702Design and orchestrate scalable batch and streaming data pipelines.
