skills.rest
genesis-agents
genesis-agentsCOMMUNITY
Explore all skills by this author →
GitHub

Data Pipeline Expert

Orchestrate end-to-end data pipelines with ingestion, deduplication, and quality checks.

2|3|Updated Nov 9, 2025
Search Tags:#data#orchestration#quality#pipeline#ingestion#deduplication
One-click install
npx skills add https://github.com/genesis-agents/GenesisPod --skill data-pipeline-expert
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Data Pipeline Expert
Source: https://github.com/genesis-agents/GenesisPod/tree/main/.claude/skills/data/data-pipeline-expert
Command: npx skills add https://github.com/genesis-agents/GenesisPod --skill data-pipeline-expert

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Unified data collection, quality management, and pipeline orchestration. Trigger keywords: data pipeline, crawler, deduplication, data quality, etl Not for: Database schema (-> schema-architect), API endpoints (-> api-developer)

Core Features & Use Cases

  • Automated data collection from diverse sources with deduplication and quality checks.
  • End-to-end pipeline orchestration from ingestion to storage, with clear handoff points to other skills.
  • Use case: When consolidating data from multiple RSS/HN sources into a unified dataset with lineage.

Quick Start

Configure a daily ingestion workflow that crawls sources, deduplicates items, applies quality checks, and stores the clean data.

Frequently Asked Questions about Data Pipeline Expert

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate an end-to-end data pipeline from ingestion to storage?▼

Data pipeline orchestration automates the flow from multi-source ingestion to storage with clear handoff points. You configure workflows that coordinate collection, processing, and storage tasks to ensure reliable end-to-end pipeline execution.

What is the best way to automate data collection from multiple sources with deduplication?▼

Automated data collection from diverse sources uses built-in deduplication and quality checks to consolidate datasets. This approach unifies multiple RSS or HN sources into a single clean dataset while maintaining data lineage.

Can I use this pipeline orchestration for daily ingestion workflows with quality monitoring?▼

Pipeline orchestration supports daily ingestion workflows that crawl sources, deduplicate items, and apply quality checks. This automated monitoring ensures clean data is stored consistently across collection and processing tasks.

Does this data pipeline skill handle database schema design or API endpoint development?▼

Data pipeline orchestration focuses on ingestion, deduplication, and quality monitoring, not database schema or API endpoints. Schema design requires a dedicated schema-architect tool, while API endpoints need an api-developer approach.

When do I need to set up structured boundaries and handoffs in a data pipeline?▼

Structured boundaries and clear handoffs are necessary when extending pipeline orchestration to downstream systems. These integration points ensure reliable transitions between collection, processing, and storage tasks across related skills.

Related Skills

Explore More Data & Analytics Skills→
cas-bigdatalabcas-bigdatalab
541

source_stop

Configure the initial data source for a DAG workflow with a file path.

Official
Basic
miptah21miptah21
2

tracing-downstream-lineage

Trace downstream table and DAG dependencies to generate change impact reports.

Community
Intermediate
usermaumusermaum

backend-engineering

Create FastAPI endpoints and SQLAlchemy models using Clean Architecture patterns.

Community
Advanced
luokai25luokai25

data-engineer

Design data pipelines, ETL/ELT processes, and data platform architectures.

Community
Advanced