language_id_score_filter
OfficialFilter and retain text samples from specified languages with high confidence scores.
Data & Analytics#text processing#language detection#confidence scoring#text filtering#sample selection
Authorcas-bigdatalab
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill identifies and filters out text samples in specified languages, based on confidence scores from a language identification process.
Core Features & Use Cases
- Language Identification and Filtering: Determines the language of each text sample and filters based on specified criteria.
- Customizable Language and Confidence Levels: Users can select specific languages and define minimum confidence thresholds for filtering.
- Use Case: Useful for analyzing multi-lingual content, such as customer support tickets, or for processing international market data, ensuring the content matches desired language requirements.
Quick Start
Apply the 'language_id_score_filter' to filter out English text with a minimum confidence score of 0.9 from the 'input_data.jsonl' file into 'output_data.jsonl'.
Dependency Matrix
Required Modules
data_juicer
Components
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: language_id_score_filter Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#language-id-score-filter Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.