ais-deduplication

Deduplicate AIS messages via Redis, Bloom Filter, and PostgreSQL pipeline.

1|Updated Feb 5, 2026

Search Tags:#real-time #PostgreSQL #Redis #deduplication #data-processing #AIS #Bloom filter

Ready to run

npx skills add https://github.com/victorhramos-dev/ais_ai --skill ais-deduplication

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ais-deduplication
Source: https://github.com/victorhramos-dev/ais_ai/tree/main/.claude/skills/ais-deduplication
Command: npx skills add https://github.com/victorhramos-dev/ais_ai --skill ais-deduplication

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill implements a three-layer deduplication strategy to prevent processing duplicate AIS messages during ingestion and analysis.

Core Features & Use Cases

Layer 1: Redis SET NX with a 60-second TTL to rapidly reject duplicates in volatile memory.
Layer 2: Bloom Filter (ReBloom) for probabilistic pre-filtering to minimize database hits.
Layer 3: PostgreSQL UPSERT to guarantee deterministic persistence by inserting only unique messages.
Hashing Window: Messages are hashed within a 30-second window to group near-similar events.
Use Case: In high-volume AIS streams from multiple sources, this stack ensures each unique AIS message is stored once across the system.

Quick Start

Configure your AIS ingestion to route messages through this three-layer deduplication workflow. Ensure Redis is running with a 60-second TTL, initialize the Bloom Filter, and connect to PostgreSQL for final persistence.

Frequently Asked Questions about ais-deduplication

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deduplicate high-throughput AIS messages in real-time?▼

Deduplicate high-throughput AIS messages in real-time by routing them through a three-layer pipeline: Redis SET NX for rapid volatile memory rejection, a Bloom Filter for probabilistic pre-filtering, and PostgreSQL UPSERT for deterministic persistence.

When should I use a Bloom filter vs Redis for maritime data deduplication?▼

Use Redis for immediate duplicate rejection with a 60-second TTL, and use a Bloom Filter for probabilistic pre-filtering to minimize database hits. The Bloom Filter serves as a secondary check before the final PostgreSQL UPSERT guarantees deterministic persistence.

How does the 30-second hashing window work for AIS stream processing?▼

The 30-second hashing window groups near-similar AIS events by hashing messages within that timeframe. This ensures recurring similar maritime data points are caught and deduplicated before hitting the PostgreSQL persistence layer.

What is the best way to prevent duplicate AIS data from multiple ingestion sources?▼

Prevent duplicate AIS data from multiple sources by applying a three-layer deduplication strategy. Redis rapidly rejects duplicates, a Bloom Filter probabilistically pre-filters, and PostgreSQL UPSERT ensures each unique message is stored exactly once across the system.

Do I need Redis and PostgreSQL configured before setting up AIS deduplication?▼

Yes, you need Redis running with a 60-second TTL for volatile memory checks and PostgreSQL connected for final persistence. The Bloom Filter must also be initialized to handle the probabilistic pre-filtering layer correctly.

Why does my AIS deduplication pipeline miss near-similar events?▼

Near-similar events are missed if the hashing window is not properly configured. The pipeline relies on a 30-second hashing window to group similar AIS events before applying the Redis, Bloom Filter, and PostgreSQL UPSERT checks.