pipeline-design-doc

Document data pipeline architectures with data flows, contracts, and governance artifacts.

19|3|Updated Feb 28, 2026

Search Tags:#design #governance #pipeline #etl #data-flow #data-contracts

Ready to run

npx skills add https://github.com/qa-aman/claude-skills --skill pipeline-design-doc

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pipeline-design-doc
Source: https://github.com/qa-aman/claude-skills/tree/main/skills/by-role/data-engineer/pipeline-design-doc
Command: npx skills add https://github.com/qa-aman/claude-skills --skill pipeline-design-doc

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Documentation of data pipelines provides a single source of truth for data movement, processing, and reliability, enabling teams to design, review, and operate pipelines confidently without digging through code.

Core Features & Use Cases

End-to-end pipeline documentation including source, ingestion, transforms, and serving layers.
Data contracts, SLAs, failure modes, and governance artifacts to improve observability and collaboration.
Templates and guidance for onboarding new engineers and conducting architecture reviews.

Quick Start

Create a design doc that captures overview, data contracts, data flows, and failure recovery, then circulate it for review.

Frequently Asked Questions about pipeline-design-doc

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I document data pipeline architectures for ETL and streaming workloads?▼

Document data pipeline architectures by capturing source, transformations, destinations, SLAs, and data governance artifacts. This creates a single source of truth for batch and streaming data movement, processing, and reliability.

What are data contracts and how do they apply to data flow governance?▼

Data contracts are enforced agreements within pipeline design docs that define data structure, quality, and ownership. They apply to data flow governance by standardizing transformation logic, failure modes, and backfill procedures for reliable processing.

How do I create a design doc for failure recovery and backfill procedures in a data pipeline?▼

Create a pipeline design doc that explicitly captures failure modes, recovery strategies, and backfill procedures. Documenting these elements ensures teams can operate pipelines confidently and restore data flows without digging through code.

Does this pipeline documentation approach work for both batch and streaming ETL pipelines?▼

Yes, this pipeline documentation approach applies to ETL and ELT pipelines across both batch and streaming workloads. It captures end-to-end data flows from ingestion to serving layers, ensuring comprehensive architecture reviews.

What's the best way to onboard new engineers to existing data pipeline transformations?▼

The best way to onboard new engineers is providing pipeline design docs with templates and guidance. These docs capture overviews, data contracts, data flows, and failure recovery, enabling quick understanding of transformation logic.

When do I need a formal data pipeline design doc instead of relying on code comments?▼

You need a formal data pipeline design doc when managing complex ETL/ELT workloads requiring clear data contracts, SLAs, and governance artifacts. It provides a single source of truth for architecture reviews, preventing reliance on digging through code.