text_pair_similarity_filter

Name: text_pair_similarity_filter
Availability: InStock
Author: cas-bigdatalab

Filter text pairs by CLIP similarity scores with configurable thresholds.

541|171|Updated May 3, 2018

Search Tags:#data filtering #data preprocessing #text similarity #text pair analysis #CLIP model

Ready to run

npx skills add https://github.com/cas-bigdatalab/piflow --skill text-pair-similarity-filter

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: text_pair_similarity_filter
Source: https://github.com/cas-bigdatalab/piflow/tree/main/workspace/skills/text_pair_similarity_filter
Command: npx skills add https://github.com/cas-bigdatalab/piflow --skill text-pair-similarity-filter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires data_juicer, and includes scripts (resource) components.

What problem does it solve?

This Skill helps in filtering text pairs based on their similarity scores, enabling users to retain or discard pairs based on specified criteria.

Core Features & Use Cases

Text Pair Similarity: Computes and filters text pairs based on their similarity using a specified CLIP model.
Flexible Filtering: Allows users to set minimum and maximum similarity scores, as well as a filtering strategy (any or all pairs must match).
Use Case: Ideal for data cleaning and preprocessing tasks, such as identifying duplicates or similar entries in large datasets.

Quick Start

Run the text_pair_similarity_filter skill with the following command:

python scripts/run_text_pair_similarity_filter.py \
  --input_path /path/to/input.jsonl \
  --output_path /path/to/output.jsonl \
  --text_key_second target_text \
  --min_score 0.85 \
  --max_score 0.99 \
  --any_or_all any

Frequently Asked Questions about text_pair_similarity_filter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I filter text pairs by similarity scores for data preprocessing?▼

You can filter text pairs by similarity scores using a specified CLIP model to compute matches and retain or discard pairs based on minimum and maximum score thresholds. This is ideal for data cleaning and preprocessing.

What is the best way to identify duplicate text entries in large datasets?▼

The best way to identify duplicates in large datasets is computing text pair similarity with a CLIP model. Setting a high minimum similarity score filters out unique entries, leaving only similar text pairs for review.

Do I need the data_juicer library to compute text pair similarity?▼

Yes, you need the data_juicer library installed to process and filter text pairs. This Skill relies on data_juicer to execute the CLIP model computations and apply the similarity filtering logic.

Can I set custom score thresholds when filtering similar text pairs?▼

Yes, you can set custom minimum and maximum similarity scores to define your filtering range. You can also specify a filtering strategy to determine whether any or all text pairs must match the criteria.

How does the CLIP model evaluate text similarity for data cleaning?▼

The CLIP model evaluates text similarity by computing vector embeddings for your text pairs and calculating their similarity scores. These scores are then used to filter the data based on your specified score range.