skills.rest
librefang
librefangOFFICIAL
Explore all skills by this author →
WebsiteGitHub

data-pipeline

Automates production-grade data pipeline design and operation for Spark, Airflow, and dbt workflows.

10|7|Updated Mar 14, 2026
Search Tags:#dbt#etl#airflow#spark#data-pipelines#data-quality#elt
One-click install
npx skills add https://github.com/librefang/librefang-registry --skill data-pipeline-librefang
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-pipeline
Source: https://github.com/librefang/librefang-registry/tree/main/skills/data-pipeline
Command: npx skills add https://github.com/librefang/librefang-registry --skill data-pipeline-librefang

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data engineers struggle to design, orchestrate, and maintain reliable production-grade pipelines across batch and streaming workloads.

Core Features & Use Cases

  • End-to-end pipeline design guidance for ETL/ELT workflows using Spark, Airflow, and dbt.
  • Guidance on data quality integration, observability, and reproducibility across environments.
  • Use Case: Build a modular DAG that ingests raw data, applies transformations, and loads to the analytics warehouse with quality checks.

Quick Start

Create a minimal Airflow-based ETL pipeline that ingests data from a source, applies transformations with dbt, and loads into the data warehouse.

Frequently Asked Questions about data-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a production-grade ETL pipeline using Airflow and dbt?▼

You build production-grade ETL pipelines by creating a modular Airflow DAG that ingests raw data, applies dbt transformations, and loads the analytics warehouse while enforcing data quality gates and idempotent processing.

What is the best way to ensure data quality and observability in Spark data pipelines?▼

Ensuring data quality and observability in Spark data pipelines requires integrating quality gates and monitoring into your orchestration workflows to guarantee reproducibility and reliable transformations across environments.

Can I use this approach for both batch and streaming ELT workflows?▼

Yes, this approach supports both batch and streaming ELT workflows. It provides guidance for designing reliable pipelines using Spark, Airflow, and dbt that handle diverse data ingestion and transformation patterns.

How do I separate orchestration from computation in Airflow DAGs?▼

Separating orchestration from computation in Airflow DAGs involves using Airflow strictly for workflow scheduling and modular task management, while delegating actual processing and transformations to frameworks like Spark and dbt.

Why do I need idempotent processing and partitioning in data engineering pipelines?▼

Idempotent processing and partitioning are needed in data engineering pipelines to guarantee reproducible execution and prevent data duplication or corruption during backfills and re-runs across different environments.

Related Skills

Explore More Data & Analytics Skills→
heyAyushhheyAyushh
2

kafka-topic-setup

Creates and configures Kafka topics for the hook event pipeline.

Community
Intermediate
kyle-chalmerskyle-chalmers
11

answering-natural-language-questions-with-dbt

Answer business analytics questions by querying dbt's Semantic Layer or ad-hoc SQL.

Community
Advanced
ironkid90ironkid90

data-autocleaning

Profile and transform BigQuery and GCS data using Dataplex and Python scripts.

Community
Advanced
oborchersoborchers
28

managed-services-over-self-hosted

Compare managed and self-hosted services using decision frameworks for small teams.

Community
Intermediate