duplicate_fragment_cleaner
OfficialEfficiently clean duplicate fragments within text data.
Data & Analytics#text normalization#text cleaning#data redundancy#duplicate fragment removal#jsonl processing
Authorcas-bigdatalab
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill addresses the issue of redundant content within text data, allowing for the removal of duplicate fragments while preserving the original content.
Core Features & Use Cases
- Duplicate Fragment Removal: Identifies and removes duplicate paragraphs, sentences, or continuous fragments within a text.
- Text Preservation: Retains the first occurrence of each unique fragment.
- Similarity Threshold: Allows for the identification of slightly modified or punctuated duplicates based on a similarity threshold.
- Use Case: Ideal for data cleaning tasks where redundancy in text needs to be minimized without affecting the overall meaning.
Quick Start
Run the 'duplicate_fragment_cleaner' skill to clean duplicate fragments from your JSONL data file 'data.jsonl' and output the results to 'cleaned.jsonl'.
Dependency Matrix
Required Modules
None requiredComponents
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: duplicate_fragment_cleaner Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#duplicate-fragment-cleaner Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.