character_repetition_filter

What problem does it solve?

This Skill solves the issue of filtering text samples based on the character-level n-gram repetition ratio, which helps users manage and clean repetitive content efficiently.

Core Features & Use Cases

• Character-level Repetition Filtering: Identifies and filters text samples where character-level n-gram repetition ratios fall outside of a specified range.
• Use Case: Ideal for cleaning up excessively repetitive text, improving data quality, and preparing text for analysis by removing redundancy.

Quick Start

Run the skill to filter text in the 'input.json' file, excluding characters with a repetition ratio between 0.0 and 0.4.