data-engineering

Name: data-engineering
Availability: InStock
Author: telagod

Orchestrates batch and streaming data pipelines with quality validation.

238|30|Updated Feb 2, 2026

Search Tags:#etl #airflow #workflow-orchestration #data-pipelines #data-engineering #dagster #stream-processing

Ready to run

npx skills add https://github.com/telagod/code-abyss --skill data-engineering-telagod

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/telagod/code-abyss/tree/main/skills/domains/data-engineering
Command: npx skills add https://github.com/telagod/code-abyss --skill data-engineering-telagod

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

数据工程域面临的数据管道编排复杂、任务依赖难以追踪以及数据质量保障缺失等挑战。本技能域提供统一的视角与方法，帮助团队规范化数据管道的设计、调度与质量控制。

Core Features & Use Cases

跨框架编排：支持 Airflow、Dagster、Prefect 的协同工作流管理
资产与依赖建模：定义数据资产、依赖关系和执行图
流式与批处理：覆盖从实时流处理到定时批处理的端到端场景
数据质量保障：集成常用数据质量工具与校验

Quick Start

Describe a typical data pipeline and trigger a basic orchestration task.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I orchestrate data pipelines across Airflow, Dagster, and Prefect?▼

Data pipeline orchestration across Airflow, Dagster, and Prefect is managed through unified workflow modeling. It coordinates cross-framework协同 tasks to enable reliable ingestion, scheduling, and dependency tracking.

Can I handle both stream processing and batch workflows in the same data pipeline?▼

Yes, data pipelines support both stream processing and batch workflows. The orchestration covers end-to-end scenarios from real-time streaming to scheduled batch processing within a single configured environment.

How do I add data quality validation checks to my ETL pipelines?▼

Data quality validation is integrated directly into ETL pipelines through workflow orchestration. It applies quality checks and guardrails during task execution to ensure reliable data ingestion and processing.

What is the best way to model data assets and track task dependencies in a workflow?▼

Modeling data assets and tracking dependencies relies on defining execution graphs within the orchestration framework. This approach maps relationships to ensure reliable workflow scheduling and governance.

Does this data pipeline orchestration approach support scalable configuration for large workflows?▼

Yes, data pipeline orchestration supports scalable configuration for large workflows. It applies guardrails and dependency tracking to maintain reliability across complex batch and streaming scenarios.