word_repetition_filter
OfficialControl text repetition ratios with n-gram filters, optimize text content.
Authorcas-bigdatalab
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill addresses the need for managing word-level n-gram repetition ratios in text content, ensuring samples are within a specific range.
Core Features & Use Cases
- N-gram Repetition Ratio Control: Filters text to retain samples within defined n-gram repetition ratios.
- Use Case: When processing large texts, this Skill is helpful for managing and controlling redundancy in content, improving readability and relevance.
Quick Start
Execute the skill to filter text repetitions for the input file 'example.txt', using the output file 'filtered_example.txt'.
Dependency Matrix
Required Modules
py-data-juicer
Components
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: word_repetition_filter Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#word-repetition-filter Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.