deduplication-logic-checker

It checks for the existence of guards against duplicate URL, title, and source identity before data insertion within the rss-to-signals and feed-orchestrator ingestion paths. This prevents data redundancy during feed ingestion and signal creation.

It detects direct insert paths lacking duplicate checks by validating the rss-to-signals and feed-orchestrator ingestion paths. It specifically identifies instances where data might be inserted without proper duplicate checks.

Feed deduplication logic validation verifies that feed ingestion and signal creation processes have checks preventing duplicate URLs, titles, or source identities before data insertion. It ensures robust logic and prevents data redundancy.

While it targets rss-to-signals and feed-orchestrator functions, you can use it whenever modifying any feed ingestion pipeline to automatically verify that no new direct insert paths without deduplication guards are introduced.

Your pipeline likely allows duplicates because direct insert paths exist without proper duplicate checks. This checker identifies those missing guards against duplicate URL, title, and source identity before data insertion.