What problem does it solves?
This Skill manages complex, version-tracked transformations of archived data, ensuring data consistency and allowing for efficient updates when underlying data models or processing logic change. It automates the tedious and error-prone process of manually reprocessing large datasets.
Core Features & Use Cases
- Version-Tracked Transformations: Automatically reprocesses only stale archives based on semantic versioning, saving significant time and compute resources by skipping unchanged data.
- Pluggable Transformers: Easily define and integrate new data transformation logic using flexible design patterns, eliminating copy-paste and promoting modularity.
- Incremental Processing: Skips unchanged archives, making subsequent runs extremely fast (e.g., from hours to seconds), ensuring rapid iteration and deployment.
- Use Case: Your data schema for archived YouTube videos changes, requiring all existing video metadata to be flattened and re-weighted. This skill automatically identifies which archives need updating, applies the new transformation, and stores the result, all while tracking versions and minimizing manual intervention.
Quick Start
To reprocess Qdrant metadata for 10 archives in a dry run:
uv run python tools/scripts/reprocess_qdrant_metadata.py --dry-run --limit 10
To run a full reprocessing:
uv run python tools/scripts/reprocess_qdrant_metadata.py
