mage-ai-pipelines

What problem does it solve?

Mage AI pipeline projects become hard to standardize when teams need consistent block design, configuration management, and reliable orchestration for both batch and streaming workloads.

Core Features & Use Cases

• Block-based pipeline composition: build pipelines from dedicated block types (data loader, transformer, exporter, sensor, custom) with clear upstream wiring.
• Hybrid SQL + Python transformations: enrich datasets using SQL blocks that reference upstream DataFrame outputs and pipeline variables.
• Operational orchestration: configure triggers (schedule/event/API), support backfills by execution date ranges, and run streaming workflows like Kafka → transform → sink.
• Environment-ready connectivity: manage database/object-storage/warehouse credentials via io_config.yaml profiles (e.g., PostgreSQL, S3, BigQuery) and attach callbacks for audit/alerts.
• Scale options: run executor_type=local_python for fast iteration or executor_type=spark for Spark workloads and Delta table reads/writes.

Quick Start

Ask the agent to generate a Mage AI pipeline named orders_daily_etl with io_config.yaml profiles, triggers.yaml cron schedule, a sensor-backed ETL flow, and a streaming Kafka pipeline template.