text_pair_similarity_filter

What problem does it solve?

This Skill helps in filtering text pairs based on their similarity scores, enabling users to retain or discard pairs based on specified criteria.

Core Features & Use Cases

• Text Pair Similarity: Computes and filters text pairs based on their similarity using a specified CLIP model.
• Flexible Filtering: Allows users to set minimum and maximum similarity scores, as well as a filtering strategy (any or all pairs must match).
• Use Case: Ideal for data cleaning and preprocessing tasks, such as identifying duplicates or similar entries in large datasets.

Quick Start

Run the text_pair_similarity_filter skill with the following command:

python scripts/run_text_pair_similarity_filter.py \
  --input_path /path/to/input.jsonl \
  --output_path /path/to/output.jsonl \
  --text_key_second target_text \
  --min_score 0.85 \
  --max_score 0.99 \
  --any_or_all any