skills.rest
expanso-io
expanso-ioOFFICIAL
Explore all skills by this author →
GitHub

expanso-fan-out-pattern

Routes a single data stream to multiple destinations including Kafka, AWS S3, Elasticsearch, and local files with fallback to file storage.

1|Updated Feb 1, 2026
Search Tags:#streaming#pipeline#data distribution#data routing#fan-out#message broker
One-click install
npx skills add https://github.com/expanso-io/expanso-skills --skill expanso-fan-out-pattern
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: expanso-fan-out-pattern
Source: https://github.com/expanso-io/expanso-skills/tree/main/skills/recipes/fan-out-pattern
Command: npx skills add https://github.com/expanso-io/expanso-skills --skill expanso-fan-out-pattern

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill efficiently distributes a single stream of incoming data to multiple destinations simultaneously, ensuring data availability across various systems.

Core Features & Use Cases

  • Multi-destination Routing: Send data to file, Kafka, S3, and Elasticsearch outputs concurrently.
  • Fault Tolerance: Implements fallback mechanisms to file storage if primary destinations are unavailable.
  • Use Case: Process real-time sensor data and simultaneously log it to a local file for debugging, stream it to Kafka for real-time analytics, archive it in S3, and index it in Elasticsearch for searching.

Quick Start

Run the fan-out pattern pipeline with sample data using the provided run script.

Frequently Asked Questions about expanso-fan-out-pattern

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I route a single data stream to multiple destinations like S3 and Kafka?▼

Routing a single data stream to multiple destinations is achieved by concurrently distributing incoming data to local files, Kafka, AWS S3, and Elasticsearch. The pipeline handles dynamic endpoint configuration and batching for efficient data flow management.

What happens if my Kafka or Elasticsearch output destination goes down during streaming?▼

If primary destinations become unavailable, the fan-out pipeline implements built-in fallback mechanisms to local file storage. This ensures enhanced reliability and prevents data loss during distribution.

Can I send real-time sensor data to Elasticsearch and S3 at the same time?▼

Yes, sending real-time sensor data to Elasticsearch and S3 simultaneously is supported. The pipeline handles multi-destination routing, allowing you to archive data in S3 and index it in Elasticsearch concurrently.

How do I configure batching for data distribution to multiple outputs?▼

Batching for data distribution is configured dynamically within the pipeline endpoints. This supports efficient data flow management when routing streams to multiple destinations simultaneously.

What is a fan-out pattern in data streaming pipelines?▼

A fan-out pattern in data streaming pipelines distributes a single incoming data stream to multiple output destinations simultaneously. This ensures data availability across various systems like file storage, Kafka, S3, and Elasticsearch.

Related Skills

Explore More Software Engineering Skills→
ajmeyers42ajmeyers42
1

wind-streams

Configure Kibana Streams and ingest pipelines for demo environments.

Community
Advanced
expanso-ioexpanso-io

expanso-kafka-to-s3

Stream data from Kafka topics to S3 buckets with partitioning and batching.

Official
Intermediate
victorzhukvictorzhuk

go-messaging

Guide Go messaging implementations across Kafka, RabbitMQ, and NATS.

Community
Intermediate
alsk1992alsk1992
650

integrations

Manage REST APIs, WebSockets, and webhooks for trading bot data streams.

Community
Advanced