skills.rest
dadbodgeoff
dadbodgeoffCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

deduplication

Deduplicate multi-source events using hash-based grouping and reputation scoring.

783|62|Updated Jan 19, 2026
Search Tags:#deduplication#data aggregation#id-based#content-based#canonical selection#reputation scoring
One-click install
npx skills add https://github.com/dadbodgeoff/drift --skill deduplication-dadbodgeoff
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: deduplication
Source: https://github.com/dadbodgeoff/drift/tree/main/drift%20v1%20depreciated/skills/deduplication
Command: npx skills add https://github.com/dadbodgeoff/drift --skill deduplication-dadbodgeoff

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill tackles the challenge of redundant data entries arising from multiple sources, ensuring data integrity and reducing noise.

Core Features & Use Cases

  • ID-Based Deduplication: Handles duplicate entries when unique identifiers are present, using a preference function to select the best record.
  • Content-Based Deduplication: Groups entries by semantic similarity and selects a canonical version based on source reputation and content quality.
  • Use Case: Aggregating news articles from various outlets to present a single, authoritative version of each story, removing redundant reports.

Quick Start

Use the deduplication skill to process a list of articles from multiple sources.

Frequently Asked Questions about deduplication

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I eliminate duplicate data entries from multiple aggregation sources?▼

To eliminate duplicate data entries from multiple aggregation sources, you use canonical selection and hash-based grouping to identify matching records and retain the most authoritative version.

What's the difference between ID-based and content-based deduplication?▼

ID-based deduplication matches records using unique identifiers, while content-based deduplication groups entries by semantic similarity and selects a canonical version using source reputation scoring.

How do I select a canonical record when aggregating news articles from various outlets?▼

You select a canonical record during news aggregation by applying reputation scoring to rank the source quality and content similarity, ensuring the highest-quality article represents the story.

Do I need a TypeScript environment to perform hash-based grouping for data deduplication?▼

Yes, you need a TypeScript or JavaScript environment with the crypto module to perform hash-based grouping, as this Skill requires custom logic for hashing and canonical selection.

How does reputation scoring work for multi-source data aggregation?▼

Reputation scoring works by assigning weights to different data sources, allowing the deduplication process to preferentially select and retain records from higher-reputation sources over redundant duplicates.

Related Skills

Explore More Data & Analytics Skills→
angryracoonangryracoon

data360-prepare

Prepare Salesforce Data Cloud ingestion assets for raw source data.

Community
Advanced
weirdgmeweirdgme
4

platform-engineering

Design and implement scalable platform engineering patterns for internal developer platforms.

Community
Advanced
massgenmassgen
1.1k

semtools

Search code and documents by semantic similarity using embeddings.

Official
Advanced
expanso-ioexpanso-io

expanso-deduplicate-events

Deduplicate HTTP events via content hashing and cache, outputting unique events to Kafka.

Official
Intermediate