iot-data-arch

Design scalable IoT data architectures with Pathway ETL pipelines.

Updated Nov 28, 2025

Search Tags:#streaming #iot #etl #data-governance #polyglot-persistence #data-architecture #pathway

Ready to run

npx skills add https://github.com/luanya01/naughty_etl --skill iot-data-arch

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: iot-data-arch
Source: https://github.com/luanya01/naughty_etl/tree/main/.agent/skills/iot-data-arch
Command: npx skills add https://github.com/luanya01/naughty_etl --skill iot-data-arch

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

本技能提供面向 IoT 數據架構與 Pathway 流式處理的標準化方法，協助設計可擴展的多層存儲與 ETL 流程，實現熱、暖、冷數據的分層管理與高效分析。

Core Features & Use Cases

多層存儲架構：熱層使用 ScyllaDB，暖層使用 Doris，冷層使用 RustFS；支持事件時間驅動的窗口計算與長期存儲。
Pathway ETL 指引：提供純函式轉換、模組化設計與測試友好型實踐，避免 IO 綁定的耦合。
快速入門與案例：針對 IoT 數據流的實際案例，展示如何從 Kafka 讀取、轉換並同時寫入多個系統。

Quick Start

在 Pathway 專案中定義 IoT 資料結構並實作符合熱/暖/冷分層的管線，含事件時間與晚資料容忍策略。

Frequently Asked Questions about iot-data-arch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design scalable IoT data pipelines with Pathway for real-time and historical analytics?▼

You design scalable IoT data pipelines with Pathway by applying a polyglot persistence architecture that separates hot, warm, and cold storage layers. This approach supports both real-time streaming analytics and long-term historical data processing within a single ETL workflow.

What is the best way to handle late data and event-time windowing in IoT ETL pipelines?▼

Handling late data and event-time windowing in IoT ETL pipelines requires defining specific tolerance strategies within your streaming framework. Pathway supports pure-function transformations that natively manage event-time windowing and late data arrivals without coupling to IO operations.

How do I configure hot, warm, and cold storage layers for IoT data architecture?▼

Configuring hot, warm, and cold storage layers for IoT data architecture involves assigning ScyllaDB to the hot layer, Doris to the warm layer, and RustFS to the cold layer. This polyglot persistence strategy optimizes both event-time windowing computations and long-term data retention.

Can I use Pathway to read from Kafka and write to multiple storage systems simultaneously?▼

Yes, you can use Pathway to read IoT data streams from Kafka, apply pure-function transformations, and simultaneously write the output to multiple storage systems. This modular ETL design ensures testability while avoiding IO-bound coupling in your pipeline.

Does Pathway support pure-function transformations for modular IoT data processing?▼

Pathway supports pure-function transformations to ensure modular and test-friendly IoT data processing. By avoiding IO-bound coupling, this approach maintains clean architectural boundaries within your streaming ETL pipelines for polyglot persistence environments.