Corpus Consolidate

To consolidate source files into a centralized corpus, this skill scans specified directories and batch-copies unique files based on SHA-256 hashing. It excludes binary, lock, and build artifacts while preserving the original directory structure for a clean dataset.

SHA-256 file hashing deduplication works by scanning disparate source locations and generating a cryptographic hash for each file. Files with identical hashes are grouped and skipped during batch copying, ensuring only unique files populate the centralized corpus directory.

To prepare a codebase for RAG indexing, use this skill to aggregate relevant source files into a single, deduplicated corpus. It manages the workflow through a state-tracked pipeline that scans, estimates tokens, and batch-copies unique files.

Yes, the consolidation pipeline automatically excludes binary files, lock files, and common build artifacts. This filtering ensures that only clean, relevant source data is processed and copied into the centralized corpus directory.

The best way to track progress during file aggregation is using a structured state machine pipeline. This skill maintains data integrity and tracks the entire consolidation lifecycle from initial scanning through token estimation to final batch-processed file copying.

Yes, you can estimate tokens for source files before batch copying. The consolidation pipeline includes a token estimation stage that evaluates scanned files prior to copying them into the centralized corpus directory for further processing.