Skills
.Work. You
Rest

stopwords_filter

Official

Filter texts based on stopwords ratio to retain relevant content.

Data & Analytics#data processing#text filtering#language processing#stopwords#relevance assessment
Authorcas-bigdatalab
Version1.0.0
Installs0

System Documentation

What problem does it solve?

The stopwords_filter Skill unit addresses the need for filtering texts to maintain a desired ratio of stopwords, helping to keep the essential content while removing noise.

Core Features & Use Cases

  • Stopwords Filtering: Filters out texts with stopwords ratios below a specified minimum threshold.
  • Customizable: Allows users to define language, text key, and minimum stopwords ratio.
  • Use Case: This Skill can be used to filter chat logs, social media posts, or other text collections to improve the relevance of content for further analysis or processing.

Quick Start

Run the stopwords_filter skill on the 'chat_logs.jsonl' file to filter out samples with stopwords ratios below 0.3, outputting to 'filtered_logs.jsonl'.

Dependency Matrix

Required Modules

py-data-juicer

Components

scripts

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: stopwords_filter
Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#stopwords-filter

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.