document_deduplicator
OfficialEfficiently remove duplicate documents using MD5 hashing.
Data & Analytics#data integrity#data cleaning#document management#MD5 hashing#document deduplication
Authorcas-bigdatalab
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill efficiently identifies and removes duplicate documents, streamlining the process of data management and ensuring data integrity.
Core Features & Use Cases
- Document Deduplication: Uses MD5 hashing to identify and remove exact duplicates.
- Configurable Parameters: Allows users to specify the text key, lowercase conversion, and character ignore options.
- File Format Flexibility: Supports JSON and TXT file formats for input and output.
- Use Case: Ideal for data scientists, researchers, or anyone dealing with large document collections where redundancy needs to be minimized.
Quick Start
Run the document_deduplicator skill to remove duplicates from your input.json file and save the results to output.json.
Dependency Matrix
Required Modules
data_juicer
Components
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: document_deduplicator Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#document-deduplicator Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.