Corpus Consolidate

Consolidate project source files into a deduplicated corpus using SHA-256 hashing.

6|3|Updated Apr 12, 2026

Search Tags:#automation #pipeline #deduplication #data-management #corpus #file-system

Ready to run

npx skills add https://github.com/carson-sweet/sweetclaude --skill corpus-consolidate

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Corpus Consolidate
Source: https://github.com/carson-sweet/sweetclaude/tree/main/skills/corpus-consolidate
Command: npx skills add https://github.com/carson-sweet/sweetclaude --skill corpus-consolidate

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill solves the problem of fragmented project data by scanning multiple source directories, identifying unique files, and consolidating them into a clean, deduplicated inbox for further processing.

Core Features & Use Cases

Intelligent Deduplication: Uses SHA-256 hashing to identify and group duplicate files across disparate source locations.
Pipeline Management: Maintains a structured state machine to ensure data integrity and track progress through the consolidation lifecycle.
Use Case: When preparing a large codebase for RAG indexing or documentation analysis, use this skill to aggregate all relevant source files into a single, deduplicated corpus while preserving the original directory structure.

Quick Start

Run the corpus-consolidate skill and provide the paths to the source directories you wish to scan for consolidation.

Frequently Asked Questions about Corpus Consolidate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I consolidate fragmented source files into a centralized corpus directory?▼

To consolidate source files into a centralized corpus, this skill scans specified directories and batch-copies unique files based on SHA-256 hashing. It excludes binary, lock, and build artifacts while preserving the original directory structure for a clean dataset.

How does SHA-256 file hashing deduplication work across multiple source directories?▼

SHA-256 file hashing deduplication works by scanning disparate source locations and generating a cryptographic hash for each file. Files with identical hashes are grouped and skipped during batch copying, ensuring only unique files populate the centralized corpus directory.

How do I prepare a large codebase for RAG indexing without duplicate files?▼

To prepare a codebase for RAG indexing, use this skill to aggregate relevant source files into a single, deduplicated corpus. It manages the workflow through a state-tracked pipeline that scans, estimates tokens, and batch-copies unique files.

Does the file consolidation pipeline automatically exclude binary and build artifacts?▼

Yes, the consolidation pipeline automatically excludes binary files, lock files, and common build artifacts. This filtering ensures that only clean, relevant source data is processed and copied into the centralized corpus directory.

What is the best way to track progress when aggregating project source files?▼

The best way to track progress during file aggregation is using a structured state machine pipeline. This skill maintains data integrity and tracks the entire consolidation lifecycle from initial scanning through token estimation to final batch-processed file copying.

Can I estimate tokens for source files before batch copying them into a corpus?▼

Yes, you can estimate tokens for source files before batch copying. The consolidation pipeline includes a token estimation stage that evaluates scanned files prior to copying them into the centralized corpus directory for further processing.