What problem does it solve?
Dagster Software-Defined Assets (SDA) solve the problem of organizing data pipeline logic as explicit, dependency-tracked, materialized data objects instead of ad-hoc scripts and opaque job steps.
Core Features & Use Cases
- Asset-based pipeline structure: define persistent data objects (tables/files/models) with clear lineage between upstream and downstream assets.
- Partitioned materialization: implement daily, static, multi-dimensional, and dynamic partitions so you recompute only what’s needed.
- Automation and triggering: use sensors and schedules, plus declarative automation with AutomationCondition (eager/on_cron/on_missing) to materialize assets reliably.
- Custom IO management: control how asset outputs are stored/loaded (for example, Parquet on S3) and capture output metadata.
- Multi-asset outputs: produce multiple dependent assets from a single operation using @multi_asset.
- Resources and deployment wiring: inject external connections via ConfigurableResource and structure Definitions for deployment.
Quick Start
Activate this skill by telling your AI to generate a Dagster project that defines @asset and partitioned assets, wires a custom S3 Parquet IO manager, and adds sensors/schedules for materialization.
