skills.rest
saolalab
saolalabOFFICIAL
Explore all skills by this author →
GitHub

Data Pipeline Engineering

Design ETL pipelines for batch and incremental data ingestion.

37|1|Updated Mar 17, 2026
Search Tags:#monitoring#migration#etl#data-modeling#idempotency#data-quality#data-pipeline
One-click install
npx skills add https://github.com/saolalab/clawforce --skill data-pipeline-engineering-saolalab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Data Pipeline Engineering
Source: https://github.com/saolalab/clawforce/tree/main/marketplace/roles/data-analyst/workspace/skills/data-pipeline
Command: npx skills add https://github.com/saolalab/clawforce --skill data-pipeline-engineering-saolalab

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides a practical, end-to-end guide for designing, building, and operating reliable data pipelines so teams can move, transform, and serve data for analytics and applications without repeated manual intervention or data-quality regressions.

Core Features & Use Cases

  • Extract–Transform–Load (ETL) Checklist: Step-by-step extraction methods, transformation validations, and load strategies to ensure consistent, repeatable pipeline runs.
  • Data Modeling Patterns: Guidance on star, snowflake, and denormalized one-big-table approaches to support OLAP workloads and query performance trade-offs.
  • Monitoring, Alerts & Runbooks: Freshness, completeness, accuracy, performance, and error monitoring recommendations with runbook and rollback procedures for operational safety.
  • Migration and Production Playbooks: Pre-migration validation, execution steps, post-migration verification, and rollback planning to safely move data between systems.
  • Best Practices: Idempotency, incremental processing, CDC, testing, and error-handling patterns for robust pipelines.

Quick Start

Design an ETL pipeline to incrementally extract sales data from the source database, apply cleaning and validation rules, load into a star-schema analytics warehouse, and configure freshness and error alerts.

Frequently Asked Questions about Data Pipeline Engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an ETL pipeline for incremental data extraction?▼

To design an incremental ETL pipeline, you apply CDC and partitioning strategies to extract changes from APIs, databases, or files, then validate and load them into your analytics system without reprocessing the full dataset.

What is the best way to monitor data quality and freshness in a data pipeline?▼

The best way to monitor data pipeline quality is by configuring alerts for freshness, completeness, and accuracy, paired with runbooks and rollback procedures to handle errors and operational anomalies safely.

How do I ensure idempotent processing in batch and incremental data pipelines?▼

Idempotent processing ensures repeated pipeline runs produce the same output without duplication, achieved through specific extraction methods, transformation validations, and load strategies that support safe retries.

When do I need star schema data modeling for my ETL pipeline?▼

You need star schema data modeling when building ETL pipelines for OLAP workloads, as it supports query performance trade-offs compared to snowflake or denormalized one-big-table approaches.

What are the steps for a database migration with pre and post validation?▼

A database migration requires pre-migration validation, execution steps, and post-migration verification, followed by rollback planning to safely move data between systems while maintaining integrity.

Can I use this approach for both batch ingestion and API extraction?▼

Yes, this approach supports both batch and incremental ingestion from APIs, databases, and files, applying data validation, schema design, and error-handling patterns to serve reporting and BI scenarios.

Related Skills

Explore More Data & Analytics Skills→
ToolsaiToolsai
108

notebooklm-studio

Automate NotebookLM research workflows for source ingestion and artifact generation.

Community
Advanced
modus-bimodus-bi

data-analyst-expert

Analyze BI datasets to generate Level-4 insights with quantified recommendations.

Official
Advanced
CatsofsufferingCatsofsuffering
5

data-engineering

Manage and validate data pipelines and streaming workflows across orchestration and quality tools.

Community
Advanced
nrakownrakow
1

reverse-etl

Design and implement reverse ETL pipelines that push warehouse data into operational tools.

Community
Advanced