Transfer data between Kafka topics and file storage in Parquet, JSON, and CSV formats.
npx skills add https://github.com/starlake-ai/starlake --skill kafkaload
Please help me install this Agent Skill. Skill: kafkaload Source: https://github.com/starlake-ai/starlake/tree/main/.agent/skills Command: npx skills add https://github.com/starlake-ai/starlake --skill kafkaload
The Kafka Load Skill solves the challenge of moving data between Apache Kafka and file-based storage, enabling seamless ingestion, offloading, and archival workflows.
Run starlake kafkaload with your configuration to transfer data between Kafka topics and file storage.
High-intent search queries and answers about installing and using this skill.
You can offload Kafka streaming data to Parquet by configuring a data transfer process to persist topic messages directly to your file system. This enables seamless ingestion and archiving workflows for downstream analytics.
Yes, you can ingest data from file storage into Kafka topics using configurable formats including JSON and CSV. The transfer supports both batch and streaming modes to move file-based data into your Kafka pipelines.
Moving data between Kafka and file storage is best handled by a dedicated transfer process that supports configurable connections, formats, and streaming modes. This approach applies to both batch ingestion and topic offloading use cases.
Yes, Kafka data transfer supports both streaming and batch modes. You can configure the transfer process to handle real-time streaming data or execute batch ingestion and archival workflows depending on your processing needs.
You configure connections and transformations for Kafka to file transfer by setting options like config, connectionRef, format, path, write-mode, stream, and transform. These settings define how data is read, processed, and written to storage.
You should offload Kafka topics to your data lake when streaming data needs to be persisted for batch processing and analytics. This archival workflow transfers messages to formats like Parquet, JSON, or CSV for downstream consumption.
nrminorQuery Parquet, CSV, and JSON files in place with DuckDB SQL.
jluo41Convert CSV, XML, Parquet, and JSON files into standardized SourceSet assets.
AbsolutelySkilledGuide building and operating Kafka and Flink streaming pipelines with exactly-once semantics.
expanso-ioStream data from Kafka topics to S3 buckets with partitioning and batching.
