skills.rest
rohunvora
rohunvoraCOMMUNITY
Explore all skills by this author →
𝕏@rohunvoraGitHub

incremental-fetch

Fetch paginated API data with two-watermark resumable progress and duplicate prevention.

20|1|Updated Jan 1, 2026
Search Tags:#api#resilience#pagination#data-ingestion#incremental-fetch#watermarks
One-click install
npx skills add https://github.com/rohunvora/cool-claude-skills --skill incremental-fetch
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: incremental-fetch
Source: https://github.com/rohunvora/cool-claude-skills/tree/main/skills/incremental-fetch
Command: npx skills add https://github.com/rohunvora/cool-claude-skills --skill incremental-fetch

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Build data pipelines that never lose progress and never re-fetch existing data. Use when fetching paginated data from external APIs and need to track progress, avoid duplicates, handle rate limits, and support both incremental updates and backfills.

Core Features & Use Cases

  • The Two Watermarks Pattern: Track two cursors (newest_id and oldest_id) to enable forward and backward fetching for incremental updates and backfills.
  • Data vs Watermark Saving: Save data pages after each fetch for resilience; commit watermarks only at end for correctness.
  • Backfill and Resume: Supports backfill mode and seamless resumption after interruptions across paginated APIs.

Quick Start

Run incremental-fetch to fetch paginated API data, saving each page and updating watermarks only after a successful run.

Frequently Asked Questions about incremental-fetch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fetch paginated API data without duplicates or losing progress?▼

Fetch paginated API data without duplicates by using a two-watermark pattern that tracks newest_id and oldest_id, saving data pages after each fetch and committing watermarks only at the end of a successful run.

What is the two-watermark pattern for incremental data ingestion?▼

The two-watermark pattern tracks two cursors, newest_id and oldest_id, enabling forward fetching for incremental updates and backward fetching for backfills across APIs that provide oldest or newest identifiers.

How do I resume an API data backfill after an interruption?▼

Resume an interrupted API backfill seamlessly by relying on the two-watermark pattern, which saves data after each page and commits progress only at the end, preventing data loss during interruptions.

Can I handle API rate limits and retries during paginated data ingestion?▼

Yes, incremental data ingestion includes built-in retry and rate-limit handling to ensure reliable fetching of paginated API data across long-running integrations without failing.

Does incremental fetching work for social feeds and price histories?▼

Yes, incremental fetching works for social feeds and price histories by tracking oldest and newest identifiers, supporting both incremental updates and backfill scenarios across paginated APIs.

When should I use a two-watermark pattern instead of basic pagination?▼

Use a two-watermark pattern instead of basic pagination when building long-running data pipelines that require resumable progress, duplicate prevention, and support for both incremental updates and backfills.

Related Skills

Explore More Software Engineering Skills→
MearmanMearman
3

wayback-range

Query the CDX API for the oldest and newest Wayback snapshots of a URL.

Community
Intermediate
hmzainjamilhmzainjamil
1

data-pipeline

Build automated ETL pipelines extracting, transforming, and loading data across systems.

Community
Advanced
riftzen-bitriftzen-bit

springboot-security

Identify and remediate security misconfigurations in Java Spring Boot services.

Community
Advanced
khokskhoks
2

strategy_trailing_stop_v2

Automate full position exits when prices fall below a configurable trailing stop threshold.

Community
Intermediate