skills.rest
shafibabar
shafibabarCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-pipeline-design

Design data pipeline topologies with processing modes, transformation strategies, and fault-tolerance mechanisms.

1|Updated Jun 20, 2026
Search Tags:#streaming#orchestration#observability#pipeline#fault-tolerance#data-architecture
One-click install
npx skills add https://github.com/shafibabar/SDLC-Artifact-Factory --skill data-pipeline-design-shafibabar
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-pipeline-design
Source: https://github.com/shafibabar/SDLC-Artifact-Factory/tree/main/skills/data-pipeline-design
Command: npx skills add https://github.com/shafibabar/SDLC-Artifact-Factory --skill data-pipeline-design-shafibabar

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill addresses the complexity of designing data pipelines by providing a structured framework for topology, fault tolerance, and observability, preventing common pitfalls like silent data loss or unscalable architectures.

Core Features & Use Cases

  • Topology Decision Framework: Guides the selection of processing modes (batch vs. streaming) and transformation strategies (ETL vs. ELT vs. streaming-transform) based on business latency requirements.
  • Fault-Tolerance Design: Defines rigorous standards for idempotency, checkpointing, and replayability to ensure data integrity under failure.
  • Use Case: Use this skill when defining a new data flow between systems to ensure the design accounts for tenant isolation, SLA targets, and observability signals from the start.

Quick Start

Use the data-pipeline-design skill to draft a topology and fault-tolerance contract for the new customer ingestion flow.

Frequently Asked Questions about data-pipeline-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design a data pipeline topology for streaming and batch processing?▼

Designing a data pipeline topology requires evaluating processing modes like batch versus streaming and selecting transformation strategies such as ETL or ELT based on your specific business latency requirements.

What is the best way to ensure fault tolerance and prevent silent data loss in data pipelines?▼

Ensuring fault tolerance in data pipelines requires defining rigorous standards for idempotent state mutation, checkpointing, and replayability to guarantee data integrity under failure conditions.

How do I architect data pipelines for end-to-end tenant isolation and specific SLA targets?▼

Architecting data pipelines for tenant isolation and SLA targets involves designing system topologies that account for event-driven choreography and observability signals from the initial design phase.

When do I need event-driven choreography and idempotent state mutation in my data architecture?▼

You need event-driven choreography and idempotent state mutation when designing data-intensive systems that require specific latency, reliability, and observability guarantees to prevent silent data loss.

Does this data pipeline design approach support both ETL and streaming-transform strategies?▼

Yes, the topology decision framework guides the selection of transformation strategies including ETL, ELT, and streaming-transform based on your business latency and processing mode requirements.

What are the limitations of building data pipelines without proper observability and checkpointing?▼

Without proper observability signals and checkpointing, data pipelines face common pitfalls like silent data loss, unscalable architectures, and an inability to replay data during failure scenarios.

Related Skills

Explore More Data & Analytics Skills→
JaviMontanoJaviMontano

metodologia-data-quality

Design a data quality framework for profiling, validation, anomaly detection, contracts, and SLA monitoring.

Community
Advanced
Prathmesh2000Prathmesh2000
1

data-pipeline

Design ETL/ELT pipelines with idempotent processing and restartable checkpoints.

Community
Advanced
LLM-Dev-OpsLLM-Dev-Ops
1

swarm-advanced

Coordinate multiple autonomous agents across research, development, and testing workflows.

Official
Advanced
ai-enhanced-engineerai-enhanced-engineer
3

gcp-cicd-patterns

Automate CI/CD pipeline design on Google Cloud Platform with Cloud Build and Cloud Deploy.

Official
Advanced