ray_document_deduplicator
OfficialEfficiently deduplicate large-scale documents using Ray distributed computing.
Authorcas-bigdatalab
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill provides a solution for efficiently removing duplicate documents from large-scale datasets, leveraging the power of Ray distributed computing for improved performance.
Core Features & Use Cases
- Document Deduplication: Removes duplicate documents based on precise matching using MD5 hash.
- Ray Distributed Framework: Utilizes the Ray distributed computing framework for parallel processing and scalability.
- Use Case: Ideal for scenarios where large volumes of documents need to be processed quickly and accurately, such as data cleaning in research or data analytics.
Quick Start
Use the ray_document_deduplicator skill to deduplicate the documents in 'input.json' and output the results to 'output.json'.
Dependency Matrix
Required Modules
rayredis
Components
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: ray_document_deduplicator Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#ray-document-deduplicator Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.