implement-incremental-extraction
OfficialGuide for building incremental metadata extraction in SQL connectors.
Authoratlanhq
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill provides expert guidance for implementing incremental metadata extraction using the Application SDK, enabling efficient updates by processing only recent changes in database schemas and tables.
Core Features & Use Cases
- Incremental Extraction Architecture: Demonstrates how to integrate incremental workflows with existing metadata pipelines.
- SQL Template Customization: Guides on creating database-specific SQL scripts for detecting table and column changes.
- State Management & Backfills: Covers handling state persistence with RocksDB and DuckDB, detecting backfill scenarios, and managing change histories.
- Use Case: Extending a new SQL database connector to support incremental updates, significantly reducing runtime and resource consumption during metadata refreshes.
Quick Start
Use this Skill to implement incremental metadata extraction for your SQL connector, focusing on customizing SQL templates and state handling logic to optimize data freshness and processing speed.
Dependency Matrix
Required Modules
None requiredComponents
references
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: implement-incremental-extraction Download link: https://github.com/atlanhq/application-sdk/archive/main.zip#implement-incremental-extraction Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.