ais-deduplication

Deduplicate high-throughput AIS messages in real-time by routing them through a three-layer pipeline: Redis SET NX for rapid volatile memory rejection, a Bloom Filter for probabilistic pre-filtering, and PostgreSQL UPSERT for deterministic persistence.

Use Redis for immediate duplicate rejection with a 60-second TTL, and use a Bloom Filter for probabilistic pre-filtering to minimize database hits. The Bloom Filter serves as a secondary check before the final PostgreSQL UPSERT guarantees deterministic persistence.

The 30-second hashing window groups near-similar AIS events by hashing messages within that timeframe. This ensures recurring similar maritime data points are caught and deduplicated before hitting the PostgreSQL persistence layer.

Prevent duplicate AIS data from multiple sources by applying a three-layer deduplication strategy. Redis rapidly rejects duplicates, a Bloom Filter probabilistically pre-filters, and PostgreSQL UPSERT ensures each unique message is stored exactly once across the system.

Yes, you need Redis running with a 60-second TTL for volatile memory checks and PostgreSQL connected for final persistence. The Bloom Filter must also be initialized to handle the probabilistic pre-filtering layer correctly.

Near-similar events are missed if the hashing window is not properly configured. The pipeline relies on a 30-second hashing window to group similar AIS events before applying the Redis, Bloom Filter, and PostgreSQL UPSERT checks.