ray_video_deduplicator
OfficialEfficiently remove duplicate videos at scale with distributed computing.
Authorcas-bigdatalab
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill addresses the challenge of identifying and removing duplicate videos, particularly in large datasets, by leveraging distributed computing with Ray and MD5 hashing.
Core Features & Use Cases
- Distributed Video Deduplication: Utilizes Ray's distributed framework to efficiently process and deduplicate large-scale video collections.
- Accuracy: Leverages MD5 hashing for precise matching of video content.
- Use Case: Ideal for data centers or content platforms that need to maintain a library of unique video content and want to eliminate redundancy.
Quick Start
Execute the skill to remove duplicates from your video collection by providing the input and output JSON file paths. Example: ray_video_deduplicator --input input.json --output output.json.
Dependency Matrix
Required Modules
rayredispy-data-juicer
Components
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: ray_video_deduplicator Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#ray-video-deduplicator Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.