skills.rest
bbvch-ai
bbvch-aiOFFICIAL
Explore all skills by this author →
GitHub

scaffold-pipeline

Generate Dagster pipeline boilerplate for external data sources and custom processing extensions.

9|2|Updated Nov 29, 2024
Search Tags:#scaffolding#rag#boilerplate#data engineering#pipeline#data ingestion#dagster
One-click install
npx skills add https://github.com/bbvch-ai/aihub-core --skill scaffold-pipeline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scaffold-pipeline
Source: https://github.com/bbvch-ai/aihub-core/tree/main/.claude/skills/scaffold-pipeline
Command: npx skills add https://github.com/bbvch-ai/aihub-core --skill scaffold-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Building new Dagster data pipelines for data ingestion or custom processing requires hours of repetitive boilerplate work, and teams frequently miss critical production-ready patterns like automation sensors, resource wiring, and proper namespace configuration that lead to avoidable production failures.

Core Features & Use Cases

  • Dual Path Support: Handles two common scenarios: connecting new external data sources (SharePoint, S3, OneDrive, etc.) to existing processing pipelines, and extending the processing framework with custom ops, resources, and IO managers.
  • Template-Driven Scaffolding: Uses pre-built source templates for common storage backends to eliminate manual configuration of rclone settings and Dagster asset definitions.
  • Production Best Practice Enforcement: Enforces consistent implementation of retry policies, resource injection, and automation conditions to avoid common Dagster pitfalls.
  • Real-World Use Case: A data engineering team needing to add a new SharePoint document library as a data source for their RAG pipeline can use this skill to generate all required boilerplate, environment variables, and Docker configuration in minutes instead of hours of manual work.

Quick Start

Use the scaffold-pipeline skill to generate a new Dagster pipeline that ingests documents from your company's SharePoint site into the existing RAG processing workflow.

Frequently Asked Questions about scaffold-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate a Dagster data pipeline for ingesting documents from SharePoint?▼

To generate a Dagster data pipeline for SharePoint ingestion, use a scaffolding tool that applies pre-built source templates. This automatically creates the required boilerplate, environment variables, and Docker configuration to connect SharePoint to your existing processing workflows.

What is the best way to add production-ready retry policies and automation sensors to a Dagster pipeline?▼

The best way to add production-ready retry policies and automation sensors to a Dagster pipeline is by using a scaffolding template that enforces these best practices. This ensures consistent resource injection and proper namespace configuration to prevent common runtime failures.

Can I use a scaffolding tool to connect S3 or OneDrive to an existing RAG processing workflow?▼

Yes, you can use a pipeline scaffolding tool to connect external data sources like S3 or OneDrive to an existing RAG processing workflow. It eliminates manual rclone configuration and generates the Dagster asset definitions needed for ingestion.

How do I extend a Dagster processing framework with custom ops and IO managers?▼

To extend a Dagster processing framework with custom ops and IO managers, use a scaffolding tool designed for dual-path support. It generates the boilerplate for custom document transformation, enrichment, and storage logic while maintaining production-ready architecture.

Why do my Dagster data pipelines keep failing due to missing namespace configuration and resource wiring?▼

Dagster data pipelines often fail due to missing namespace configuration and resource wiring because these production-ready patterns are frequently overlooked during manual boilerplate creation. Using a template-driven scaffolding approach enforces consistent implementation to avoid these runtime failures.

Do I need to manually configure rclone settings when adding SFTP as a data source in Dagster?▼

No, you do not need to manually configure rclone settings when adding SFTP as a data source in Dagster. A template-driven scaffolding tool eliminates manual configuration by using pre-built source templates for common storage backends like SFTP.

Related Skills

Explore More Software Engineering Skills→
bbvch-aibbvch-ai
9

rclone-guide

Configure rclone cloud storage connectors for Dagster data pipelines.

Official
Intermediate
PaldomPaldom
1

unstructured-pdf-generation

Generate synthetic PDF documents with JSON evaluation files for RAG systems.

Community
Intermediate
bbvch-aibbvch-ai
9

dagster-pipelines

Implement Dagster pipeline architectures with asset factories, ops, and resources.

Official
Advanced
rittmananalyticsrittmananalytics
8

dagster

Automate Dagster pipeline setup and management with dbt integration.

Official
Advanced