skills.rest
Renzo-Tognella
Renzo-TognellaCOMMUNITY
Explore all skills by this author →
GitHub

Semantic Deduplication

Detect and resolve duplicates in large datasets using multi-level similarity matching.

Updated Apr 26, 2026
Search Tags:#pipeline#duplicate detection#semantic analysis#data deduplication#vector similarity
One-click install
npx skills add https://github.com/Renzo-Tognella/UniversalThingsForMyAgents --skill semantic-deduplication
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Semantic Deduplication
Source: https://github.com/Renzo-Tognella/UniversalThingsForMyAgents/tree/main/skills/26_semantic_deduplication
Command: npx skills add https://github.com/Renzo-Tognella/UniversalThingsForMyAgents --skill semantic-deduplication

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of detecting and resolving duplicates in large datasets, particularly in scenarios where exact matches are not sufficient due to conceptual similarity.

Core Features & Use Cases

  • Multi-Level Deduplication: Offers a four-level deduplication pipeline, from exact match to conceptual similarity.
  • Vector Similarity: Incorporates vector similarity for more nuanced duplicate detection.
  • Merge and Refine Strategies: Provides strategies for merging or refining duplicates based on context and similarity scores.
  • Use Case: Ideal for TheSearch admission and consolidation stages, where resolving similar memories is crucial.

Quick Start

Use the semantic_dedup skill to deduplicate memories in your dataset, considering all levels of the deduplication pipeline.

Frequently Asked Questions about Semantic Deduplication

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect duplicates in a dataset when exact matching is not enough?▼

Semantic deduplication identifies conceptually similar records using vector similarity and semantic refinement. It resolves duplicates where exact matching fails, catching variations in phrasing or meaning across large datasets.

How does vector similarity improve data deduplication pipelines?▼

Vector similarity improves deduplication by comparing mathematical representations of data meaning rather than surface text. This enables detection of conceptual duplicates that string matching misses, refining results across large datasets.

What is the best way to merge similar memories in TheSearch admission stages?▼

The best way to merge similar memories is a four-level deduplication pipeline. It progresses from exact match and Jaro-Winkler to vector similarity and semantic refinement, merging records based on contextual similarity scores.

Can I use Jaro-Winkler for semantic analysis in a duplicate detection pipeline?▼

Yes, Jaro-Winkler is used in the pipeline to measure string similarity before applying vector similarity. It serves as an intermediate level to catch near-exact matches before deeper semantic refinement evaluates conceptual similarity.

When do I need semantic refinement for data deduplication?▼

Semantic refinement is needed when records share conceptual meaning but lack exact string overlap. It resolves conceptually similar entries that exact match and Jaro-Winkler levels cannot detect in large datasets.

Related Skills

Explore More Software Engineering Skills→
MarioDeFelipeMarioDeFelipe
25

Intelligent Lookup Wizard

Match messy text records across datasets using fuzzy and composite algorithms.

Community
Advanced
mizchimizchi
821

check-similarity

Detect duplicate code across languages using AST-based similarity tools.

Community
Advanced
betalyrabetalyra
30

effect-uai-embedding

Convert text and images into vector embeddings for semantic search and retrieval.

Official
Intermediate
zilliztechzilliztech
3

duplicate-detection

Detect exact and near-duplicate records using hash checks and semantic similarity.

Official
Advanced