flagged_words_filter

Name: flagged_words_filter
Availability: InStock
Author: cas-bigdatalab

Filters text samples with a high proportion of flagged words using customizable thresholds and multilingual support.

541|171|Updated May 3, 2018

Search Tags:#data cleaning #text analysis #content moderation #text filtering #sensitive word detection

Ready to run

npx skills add https://github.com/cas-bigdatalab/piflow --skill flagged-words-filter

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: flagged_words_filter
Source: https://github.com/cas-bigdatalab/piflow/tree/main/workspace/skills/flagged_words_filter
Command: npx skills add https://github.com/cas-bigdatalab/piflow --skill flagged-words-filter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires data_juicer, and includes scripts (resource) components.

What problem does it solve?

This Skill is designed to address the need for filtering text that contains a proportion of flagged words above a specified threshold.

Core Features & Use Cases

Text Filtering: Automatically remove samples from text that contain a high percentage of flagged words.
Multilingual Support: Supports multiple languages for versatile usage.
Use Case: It can be utilized in content moderation, data cleaning, and any scenario where the presence of sensitive words needs to be minimized.

Quick Start

Run the flagged_words_filter skill with the input file 'sample_input.json' and output to 'filtered_output.json'.

Frequently Asked Questions about flagged_words_filter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I filter text data containing a high proportion of sensitive words?▼

You can filter text data with sensitive words by removing samples where flagged words exceed a specified threshold. This Skill calculates the proportion of flagged words in each sample and automatically discards those that surpass your defined limit.

Does sensitive word detection support multiple languages for content moderation?▼

Yes, sensitive word detection supports multiple languages for versatile content moderation. It filters flagged words across different linguistic datasets, making it suitable for multilingual data cleaning tasks.

Do I need data_juicer to run text filtering for flagged words?▼

Yes, you need data_juicer to run text filtering for flagged words. The Skill requires the data_juicer framework for its core operation and processing logic to identify and remove samples with high sensitive word proportions.

What is the best way to remove text samples with a high percentage of flagged words?▼

The best way to remove text samples with a high percentage of flagged words is using a threshold-based filter. This Skill automatically drops samples exceeding your specified flagged word ratio, streamlining content moderation and data cleaning workflows.

Can I customize the filtering threshold for sensitive word detection?▼

Yes, you can customize the filtering threshold for sensitive word detection by specifying the maximum allowed proportion of flagged words. Samples exceeding this threshold are automatically removed from your dataset.