data-engineering-importer
OfficialAutomate historical data import into ClickHouse.
AuthorMosaic-agent
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Data engineers need a reliable, end-to-end workflow to ingest, govern, and repair historical market data into ClickHouse. This skill encapsulates the adapter-based fetcher pattern, delta-sync with watermarks, and schema maintenance to keep the data lake accurate over time.
Core Features & Use Cases
- Adapter-based fetchers for new data sources with automatic registration and watermark management, enabling rapid expansion of the data pipeline.
- Delta-sync with watermark tracking to ensure idempotent backfills and safe incremental updates across multiple segments and partitions.
- Backfill, validation, and repair workflows for corrupted partitions and missing data to preserve data integrity and continuity.
Quick Start
Implement a new data fetcher, register it in the registry, and run the importer CLI to ingest historical data.
Dependency Matrix
Required Modules
clickhouse_connect
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: data-engineering-importer Download link: https://github.com/Mosaic-agent/Mosaic-fund-agent/archive/main.zip#data-engineering-importer Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 510,000+ vetted skills library on demand.