dagster-pipelines

What problem does it solve?

This Skill eliminates the steep learning curve and common pitfalls of building production-grade Dagster pipelines, which require mastering asset-centric architecture, dynamic partitions, automation conditions, and integrated resource configurations for data and AI workloads.

Core Features & Use Cases

• Complete Dagster Component Reference: Covers asset factories, ops, resources, IO managers, sensors, schedules, and automation conditions with implementation examples and best practices.
• Pre-built Pipeline Patterns: Includes factory functions for two-stage document processing pipelines (SharePoint to S3 to vector store) and configurable Definitions for common data ingestion and RAG workflows.
• Use Case: A data engineering team building a knowledge base pipeline can use this Skill to correctly implement observable assets that detect new document uploads, auto-materialize embedding assets, and configure Milvus IO managers for vector storage.

Quick Start

Use the dagster-pipelines skill to set up an observable asset that automatically triggers downstream processing when new documents are added to your S3 data lake.