data-engineering-importer

Name: data-engineering-importer
Availability: InStock
Author: Mosaic-agent

Automate historical data ingestion and ClickHouse schema maintenance for market data pipelines.

36|7|Updated Feb 22, 2026

Search Tags:#etl #backfill #clickhouse #data-ingestion #watermarks #delta-sync

Ready to run

npx skills add https://github.com/Mosaic-agent/Mosaic-fund-agent --skill data-engineering-importer

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineering-importer
Source: https://github.com/Mosaic-agent/Mosaic-fund-agent/tree/main/.gravity/skills/data-engineering-importer
Command: npx skills add https://github.com/Mosaic-agent/Mosaic-fund-agent --skill data-engineering-importer

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires clickhouse_connect, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Data engineers need a reliable, end-to-end workflow to ingest, govern, and repair historical market data into ClickHouse. This skill encapsulates the adapter-based fetcher pattern, delta-sync with watermarks, and schema maintenance to keep the data lake accurate over time.

Core Features & Use Cases

Adapter-based fetchers for new data sources with automatic registration and watermark management, enabling rapid expansion of the data pipeline.
Delta-sync with watermark tracking to ensure idempotent backfills and safe incremental updates across multiple segments and partitions.
Backfill, validation, and repair workflows for corrupted partitions and missing data to preserve data integrity and continuity.

Quick Start

Implement a new data fetcher, register it in the registry, and run the importer CLI to ingest historical data.

Frequently Asked Questions about data-engineering-importer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I backfill historical market data into ClickHouse safely?▼

Delta-sync with watermarks tracks the last successfully ingested record, ensuring updates are idempotent and preventing duplicate rows during incremental ClickHouse data ingestion. This maintains data integrity across segments and partitions.

How do I add a new data source to an existing ClickHouse ingestion pipeline?▼

Implement an adapter-based fetcher for the new source, register it in the pipeline registry, and run the importer CLI. This automates historical data ingestion and ClickHouse schema maintenance for the new market data domain.

How do I validate integrity and repair corrupted partitions in ClickHouse?▼

The pipeline includes built-in validation and repair workflows to fix corrupted partitions and recover missing data. This preserves data integrity and continuity across diverse market data domains stored in ClickHouse.

Do I need the clickhouse_connect library to use this data ingestion pipeline?▼

Yes, the pipeline requires the clickhouse_connect dependency to interface with the database. This library provides the underlying connection required to execute schema maintenance, backfills, and delta-sync operations.

What is the best way to handle incremental updates for diverse market data sources?▼

Using a registry-based fetcher pattern with watermark-driven delta-sync ensures idempotent backfills and safe incremental updates. This approach scales across diverse market data domains while maintaining partition integrity.

Why does my incremental data sync result in duplicate records in ClickHouse?▼

Duplicate records during incremental sync often occur when watermarks are not properly tracked or applied. The pipeline's watermark-driven delta-sync mechanism ensures idempotent backfills and prevents duplicates across partitions.