scaffold-pipeline

What problem does it solve?

Building new Dagster data pipelines for data ingestion or custom processing requires hours of repetitive boilerplate work, and teams frequently miss critical production-ready patterns like automation sensors, resource wiring, and proper namespace configuration that lead to avoidable production failures.

Core Features & Use Cases

• Dual Path Support: Handles two common scenarios: connecting new external data sources (SharePoint, S3, OneDrive, etc.) to existing processing pipelines, and extending the processing framework with custom ops, resources, and IO managers.
• Template-Driven Scaffolding: Uses pre-built source templates for common storage backends to eliminate manual configuration of rclone settings and Dagster asset definitions.
• Production Best Practice Enforcement: Enforces consistent implementation of retry policies, resource injection, and automation conditions to avoid common Dagster pitfalls.
• Real-World Use Case: A data engineering team needing to add a new SharePoint document library as a data source for their RAG pipeline can use this skill to generate all required boilerplate, environment variables, and Docker configuration in minutes instead of hours of manual work.

Quick Start

Use the scaffold-pipeline skill to generate a new Dagster pipeline that ingests documents from your company's SharePoint site into the existing RAG processing workflow.