skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

mage-ai-pipelines

Design Mage AI pipelines with block-level composition for ETL and streaming workflows.

14|1|Updated May 5, 2026
Search Tags:#data pipelines#etl orchestration#mage-ai#hybrid sql python#streaming kafka#backfill triggers#spark delta
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill mage-ai-pipelines
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: mage-ai-pipelines
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/mage_ai
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill mage-ai-pipelines

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Mage AI pipeline projects become hard to standardize when teams need consistent block design, configuration management, and reliable orchestration for both batch and streaming workloads.

Core Features & Use Cases

  • Block-based pipeline composition: build pipelines from dedicated block types (data loader, transformer, exporter, sensor, custom) with clear upstream wiring.
  • Hybrid SQL + Python transformations: enrich datasets using SQL blocks that reference upstream DataFrame outputs and pipeline variables.
  • Operational orchestration: configure triggers (schedule/event/API), support backfills by execution date ranges, and run streaming workflows like Kafka → transform → sink.
  • Environment-ready connectivity: manage database/object-storage/warehouse credentials via io_config.yaml profiles (e.g., PostgreSQL, S3, BigQuery) and attach callbacks for audit/alerts.
  • Scale options: run executor_type=local_python for fast iteration or executor_type=spark for Spark workloads and Delta table reads/writes.

Quick Start

Ask the agent to generate a Mage AI pipeline named orders_daily_etl with io_config.yaml profiles, triggers.yaml cron schedule, a sensor-backed ETL flow, and a streaming Kafka pipeline template.

Frequently Asked Questions about mage-ai-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build ETL pipelines with hybrid SQL and Python transformations?▼

Hybrid SQL and Python ETL pipelines use dedicated block types where SQL blocks reference upstream DataFrame outputs and pipeline variables. This block-based composition provides clear upstream wiring for repeatable data transformation workflows.

Can I use Mage AI for both scheduled batch processing and Kafka streaming workflows?▼

Mage AI handles both scheduled batch processing and Kafka streaming workflows. Operational orchestration is configured via triggers for schedule, event, or API-driven execution, including backfill support by execution date ranges.

How do I configure database and warehouse credentials for Mage AI pipelines?▼

Database and warehouse credentials for Mage AI pipelines are managed through io_config.yaml connection profiles. This environment-ready connectivity supports PostgreSQL, S3, and BigQuery for secure external data source access.

What is the best way to run Mage AI pipelines on Spark with Delta table reads and writes?▼

To run Mage AI pipelines on Spark with Delta table support, set executor_type to spark. This scale option handles Spark workloads, contrasting with executor_type=local_python used for fast local iteration during pipeline development.

How do I trigger backfills and REST API executions for data pipeline orchestration?▼

Backfills and REST API executions for data pipelines are configured using YAML wiring files like triggers.yaml and metadata.yaml. These define event-driven runs, API-triggered execution, and backfills by specific execution date ranges.

Does Mage AI pipeline orchestration support dbt integration and audit callbacks?▼

Mage AI pipeline orchestration supports dbt integration for transformations and allows attaching callbacks for audit and alerts. This standardizes block design and configuration management across batch and streaming workloads.

Related Skills

Explore More Software Engineering Skills→
rbutinarrbutinar

fabric-sp-run

Execute and monitor stored procedures on a Microsoft Fabric Warehouse.

Community
Intermediate
pacphipacphi
6

n8n-trigger-testing-strategies

Diagnose and validate n8n trigger behavior across webhook, schedule, polling, and event-driven triggers.

Community
Advanced
Patasse97Patasse97

senior-data-engineer

Design and orchestrate data pipelines with Python, SQL, Spark, Airflow, and dbt.

Community
Advanced
itsadijmbtitsadijmbt
1

spark-python-data-source

Build custom PySpark DataSource connectors for batch and Structured Streaming.

Community
Advanced