fiftyone-find-duplicates

Name: fiftyone-find-duplicates
Availability: InStock
Author: Voxel51

Finds and deduplicates near-identical images in FiftyOne datasets using similarity scoring and thresholds.

37|8|Updated Dec 18, 2025

Search Tags:#workflow #brain #deduplicate #datasets #image-similarity #FiftyOne #near-duplicates

Ready to run

npx skills add https://github.com/voxel51/fiftyone-skills --skill fiftyone-find-duplicates

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: fiftyone-find-duplicates
Source: https://github.com/voxel51/fiftyone-skills/tree/main/skills/fiftyone-find-duplicates
Command: npx skills add https://github.com/voxel51/fiftyone-skills --skill fiftyone-find-duplicates

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps identify and remove near-duplicate images within FiftyOne datasets by comparing embeddings and applying similarity thresholds to group similar samples.

Core Features & Use Cases

Near-duplicate detection: Identify groups of visually similar samples within a dataset.
Deduplication workflow: Review results in the FiftyOne App and delete redundant samples while keeping representatives.
Use Case: You have a large image collection and need to curate a clean dataset for model training by removing near-duplicates.

Quick Start

Launch FiftyOne, verify the brain plugin, compute embeddings for your dataset, and run the near-duplicate search to identify and review duplicates. Then proceed to deduplicate as needed.

Frequently Asked Questions about fiftyone-find-duplicates

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I find and remove near-duplicate images in a dataset?▼

Near-duplicate detection works by computing visual embeddings for your images and comparing them using a similarity threshold. Samples scoring above the threshold are grouped, allowing you to isolate visually similar images for review and removal.

How do I deduplicate a large image collection for model training?▼

You can deduplicate an image collection by running a brain-based similarity search to group visually similar samples. After reviewing the grouped duplicates in the FiftyOne App, you delete redundant samples to ensure a clean dataset for model training.

Can I review detected near-duplicates before deleting them from my dataset?▼

Yes, detected near-duplicates are isolated into saved views for guided review in the FiftyOne App. This allows you to visually inspect grouped duplicates and selectively delete redundant samples while keeping representative images.

Do I need to compute embeddings to find near-duplicates in FiftyOne?▼

Yes, computing embeddings is a required step. The Skill uses these visual embeddings to calculate brain-based similarity scores between images, which identifies near-duplicates when evaluated against a configured threshold.

How does a similarity threshold affect near-duplicate detection?▼

The similarity threshold determines the strictness of near-duplicate grouping. A higher threshold requires images to be nearly identical, while a lower threshold groups more broadly similar samples, directly controlling the scope of the deduplication review.