skills.rest
cas-bigdatalab
cas-bigdatalabOFFICIAL
Explore all skills by this author →
WebsiteGitHub

flagged_words_filter

Filters text samples with a high proportion of flagged words using customizable thresholds and multilingual support.

541|171|Updated May 3, 2018
Search Tags:#data cleaning#text analysis#content moderation#text filtering#sensitive word detection
One-click install
npx skills add https://github.com/cas-bigdatalab/piflow --skill flagged-words-filter
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flagged_words_filter
Source: https://github.com/cas-bigdatalab/piflow/tree/main/workspace/skills/flagged_words_filter
Command: npx skills add https://github.com/cas-bigdatalab/piflow --skill flagged-words-filter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires data_juicer, and includes scripts (resource) components.

What problem does it solve?

This Skill is designed to address the need for filtering text that contains a proportion of flagged words above a specified threshold.

Core Features & Use Cases

  • Text Filtering: Automatically remove samples from text that contain a high percentage of flagged words.
  • Multilingual Support: Supports multiple languages for versatile usage.
  • Use Case: It can be utilized in content moderation, data cleaning, and any scenario where the presence of sensitive words needs to be minimized.

Quick Start

Run the flagged_words_filter skill with the input file 'sample_input.json' and output to 'filtered_output.json'.

Frequently Asked Questions about flagged_words_filter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I filter text data containing a high proportion of sensitive words?▼

You can filter text data with sensitive words by removing samples where flagged words exceed a specified threshold. This Skill calculates the proportion of flagged words in each sample and automatically discards those that surpass your defined limit.

Does sensitive word detection support multiple languages for content moderation?▼

Yes, sensitive word detection supports multiple languages for versatile content moderation. It filters flagged words across different linguistic datasets, making it suitable for multilingual data cleaning tasks.

Do I need data_juicer to run text filtering for flagged words?▼

Yes, you need data_juicer to run text filtering for flagged words. The Skill requires the data_juicer framework for its core operation and processing logic to identify and remove samples with high sensitive word proportions.

What is the best way to remove text samples with a high percentage of flagged words?▼

The best way to remove text samples with a high percentage of flagged words is using a threshold-based filter. This Skill automatically drops samples exceeding your specified flagged word ratio, streamlining content moderation and data cleaning workflows.

Can I customize the filtering threshold for sensitive word detection?▼

Yes, you can customize the filtering threshold for sensitive word detection by specifying the maximum allowed proportion of flagged words. Samples exceeding this threshold are automatically removed from your dataset.

Related Skills

Explore More Content & Communication Skills→
cas-bigdatalabcas-bigdatalab
541

clean_copyright_mapper

Remove copyright comments from the beginning of text samples using Python's data_juicer library.

Official
Basic
cas-bigdatalabcas-bigdatalab
541

words_num_filter

Filter text samples in JSON/JSONL files by word count limits.

Official
Basic
cas-bigdatalabcas-bigdatalab
541

average_line_length_filter

Filter text samples by average line length using Python scripts.

Official
Basic
phuhao00phuhao00
12

content-moderator

Detect sensitive words and verify content compliance against platform rules.

Community
Basic