skills.rest
cas-bigdatalab
cas-bigdatalabOFFICIAL
Explore all skills by this author →
WebsiteGitHub

text_pair_similarity_filter

Filter text pairs by CLIP similarity scores with configurable thresholds.

541|171|Updated May 3, 2018
Search Tags:#data filtering#data preprocessing#text similarity#text pair analysis#CLIP model
One-click install
npx skills add https://github.com/cas-bigdatalab/piflow --skill text-pair-similarity-filter
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: text_pair_similarity_filter
Source: https://github.com/cas-bigdatalab/piflow/tree/main/workspace/skills/text_pair_similarity_filter
Command: npx skills add https://github.com/cas-bigdatalab/piflow --skill text-pair-similarity-filter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires data_juicer, and includes scripts (resource) components.

What problem does it solve?

This Skill helps in filtering text pairs based on their similarity scores, enabling users to retain or discard pairs based on specified criteria.

Core Features & Use Cases

  • Text Pair Similarity: Computes and filters text pairs based on their similarity using a specified CLIP model.
  • Flexible Filtering: Allows users to set minimum and maximum similarity scores, as well as a filtering strategy (any or all pairs must match).
  • Use Case: Ideal for data cleaning and preprocessing tasks, such as identifying duplicates or similar entries in large datasets.

Quick Start

Run the text_pair_similarity_filter skill with the following command:

python scripts/run_text_pair_similarity_filter.py \
  --input_path /path/to/input.jsonl \
  --output_path /path/to/output.jsonl \
  --text_key_second target_text \
  --min_score 0.85 \
  --max_score 0.99 \
  --any_or_all any

Frequently Asked Questions about text_pair_similarity_filter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I filter text pairs by similarity scores for data preprocessing?▼

You can filter text pairs by similarity scores using a specified CLIP model to compute matches and retain or discard pairs based on minimum and maximum score thresholds. This is ideal for data cleaning and preprocessing.

What is the best way to identify duplicate text entries in large datasets?▼

The best way to identify duplicates in large datasets is computing text pair similarity with a CLIP model. Setting a high minimum similarity score filters out unique entries, leaving only similar text pairs for review.

Do I need the data_juicer library to compute text pair similarity?▼

Yes, you need the data_juicer library installed to process and filter text pairs. This Skill relies on data_juicer to execute the CLIP model computations and apply the similarity filtering logic.

Can I set custom score thresholds when filtering similar text pairs?▼

Yes, you can set custom minimum and maximum similarity scores to define your filtering range. You can also specify a filtering strategy to determine whether any or all text pairs must match the criteria.

How does the CLIP model evaluate text similarity for data cleaning?▼

The CLIP model evaluates text similarity by computing vector embeddings for your text pairs and calculating their similarity scores. These scores are then used to filter the data based on your specified score range.

Related Skills

Explore More Data & Analytics Skills→
cas-bigdatalabcas-bigdatalab
541

image_text_similarity_filter

Filter images by textual similarity using a HuggingFace CLIP model.

Official
Advanced
jackquelinunpredictable827jackquelinunpredictable827
1

safha

Scrape and clean Arabic web content into JSONL training datasets.

Community
Intermediate
cas-bigdatalabcas-bigdatalab
541

average_line_length_filter

Filter text samples by average line length using Python scripts.

Official
Basic
gqf2008gqf2008

clip

Map images and text to shared embeddings for zero-shot classification and retrieval.

Community
Intermediate