What problem does it solve?
Domino Flows solves the challenge of coordinating multi-stage machine learning and data processing pipelines across heterogeneous execution environments while preserving reproducibility, lineage, and observability.
Core Features & Use Cases
- DAG-based orchestration: Connect task stages as directed acyclic graphs so complex multi-step workflows execute in a predictable order.
- Heterogeneous task environments: Run different stages as Domino Jobs with environment and hardware overrides (CPU, GPU tiers).
- Typed inputs/outputs & artifact passing: Pass small JSON metadata via /workflow/inputs and /workflow/outputs while storing large artifacts in /mnt/artifacts.
- Observability and debugging: Monitor executions and node states via FlyteRemote and inspect task execution errors for failed nodes.
- Use Case: Build a daily ETL → feature engineering → training → evaluation pipeline where ingest and training run in different Domino environments and outputs are versioned for reproducibility.
Quick Start
Trigger the training_pipeline remotely with PYTHONPATH set to /mnt/code and the input path /mnt/data/raw.csv to run the full flow.
