claim-dedup-and-compact

What problem does it solve?

It solves the problem of repeated or duplicate claim records accumulating in an append-only JSONL store, which bloats datasets and makes downstream processing slower and less reliable.

Core Features & Use Cases

• Per-source write-time dedup: Removes duplicates during the Store phase by checking content hashes only within the target by-source file, keeping the hot path fast.
• On-demand global compaction: Deduplicates and rewrites the master and all derived indexes (by-topic and by-source) using a deterministic pattern that preserves the oldest claim.
• Atomic Windows-safe rewrite with backups: Performs atomic-ish write using temp files plus backup copies (with .bak.<timestamp>) to reduce corruption risk, and supports DryRun to compute effects without touching files.
• Debuggable response contract: Returns fields like ExtractedCount and DedupSkipped so you can assert dedup behavior in CI or during operational checks.

Use case: Your LLM extracts the same factual claim multiple times from the same evidence source; you want to ensure you store it once, and later compact the whole claim store so old duplicates are removed while keeping indices consistent.

Quick Start

Ask your system to run claim store compaction in DryRun mode first to estimate BeforeCount and AfterCount, then run the real compact once you confirm the removal count is acceptable.