Orchestrate streaming Elixir data pipelines with GenStage, Broadway, and Flow.
npx skills add https://github.com/layeddie/ai-rules --skill data-pipeline-layeddie
Please help me install this Agent Skill. Skill: data-pipeline Source: https://github.com/layeddie/ai-rules/tree/main/skills/data-pipeline Command: npx skills add https://github.com/layeddie/ai-rules --skill data-pipeline-layeddie
This skill helps teams design and implement scalable streaming data pipelines in Elixir using GenStage, Broadway, and Flow, enabling backpressure-aware processing and high-throughput transformations.
Create a GenStage producer, wire it to a Broadway pipeline, and start a simple flow of events to observe processing.
High-intent search queries and answers about installing and using this skill.
Scalable Elixir data pipelines use GenStage, Broadway, and Flow to orchestrate streaming events. They enforce backpressure handling and producer-consumer topologies, enabling reliable processing for real-time analytics and high-throughput ingestion.
Backpressure handling in Elixir streaming regulates event flow between producers and consumers to prevent overload. It is needed for ETL transformations and real-time analytics where reliable processing of logs, sensor data, or event streams is required.
To set up GenStage and Broadway, create a GenStage producer, wire it to a Broadway pipeline, and start a simple flow of events. This establishes a producer-consumer topology to observe and process streaming data.
Yes, Elixir Flow processes real-time event streams and ETL transformations by building fan-out and fan-in streaming topologies. This connects producers with multiple consumers for scalable, high-throughput data processing.
Broadway supports fan-out and fan-in streaming topologies to connect producers with multiple consumers. This architecture enables backpressure-aware processing and high-throughput transformations in Elixir.
GenStage and Broadway are designed for high-throughput, backpressure-aware streaming data processing. They are not suited for simple, synchronous tasks or scenarios not requiring producer-consumer topologies and end-to-end streaming patterns.
mkreymanRun Mix-based quality checks for Elixir/Phoenix projects.
haxlysOptimize Spark jobs with partitioning, memory tuning, and shuffle optimization.
vuralserhat86Implements asynchronous communication patterns using Kafka, RabbitMQ, NATS, Redis Streams, Celery, BullMQ, Temporal, and more.
agnaldo4jBuild parallel data pipelines in Elixir using the Flow library.
