validate-incremental-sync

Validate CDC offset tracking and record filtering in data connectors.

60|116|Updated Nov 13, 2025

Search Tags:#data pipeline #data validation #incremental sync #cdc #connector testing #offset tracking

Ready to run

npx skills add https://github.com/databrickslabs/lakeflow-community-connectors --skill validate-incremental-sync

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: validate-incremental-sync
Source: https://github.com/databrickslabs/lakeflow-community-connectors/tree/main/.claude/skills/validate-incremental-sync
Command: npx skills add https://github.com/databrickslabs/lakeflow-community-connectors --skill validate-incremental-sync

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill ensures that your data connectors accurately track changes using Change Data Capture (CDC) or incremental sync, preventing data loss or duplication.

Core Features & Use Cases

Offset Validation: Verifies that the connector correctly identifies and uses offset keys (like timestamps or IDs) to track data progression.
Filtering Accuracy: Tests if passing a specific offset value successfully filters records, ensuring only new or modified data is processed.
Use Case: After implementing a new CDC connector for a critical database table, use this Skill to confirm that it reliably captures all updates and deletes without missing any records or reprocessing old ones.

Quick Start

Validate the incremental sync behavior for the 'orders' table in the 'ecommerce' connector.

Frequently Asked Questions about validate-incremental-sync

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I validate CDC offset tracking to prevent data loss in incremental sync?▼

You validate incremental sync offset tracking by testing if passing a specific offset value successfully filters records, ensuring the connector only processes new or modified data while accurately reflecting the last processed record.

What is incremental sync record filtering and how does it work?▼

Incremental sync record filtering isolates new or modified data by passing a specific offset value to the connector. This ensures only data after the last processed record is captured, preventing duplication and data loss.

How do I test if my data connector is correctly filtering records by offset?▼

Test filtering accuracy by passing a specific offset value to your data connector and verifying it successfully isolates new or modified records. This confirms the connector correctly tracks data progression without missing updates or deletes.

Can I use this to validate change data capture for database table updates?▼

Yes, you can validate change data capture for database tables by confirming the connector reliably captures all updates and deletes. It ensures offset values accurately reflect the last processed record for critical data synchronization.

Why does my incremental sync reprocess old records or miss updates?▼

Incremental sync reprocesses old records or misses updates when offset values fail to accurately reflect the last processed record. Validating offset tracking and filtering mechanisms ensures reliable change detection and prevents data loss.

What are the limitations of relying on offset tracking for change data capture?▼

Relying on offset tracking for change data capture requires accurate offset keys like timestamps or IDs. Limitations arise if the connector fails to correctly identify these keys, leading to inaccurate filtering, missed updates, or duplicated records.