agentsop-idempotent-ingestion

Re-running RAG ingestion creates duplicates because the pipeline lacks a persisted content-hash ledger to track previously indexed documents. Implementing idempotent ingestion with a docstore-backed strategy ensures unchanged documents are skipped on subsequent runs.

To make LlamaIndex ingestion idempotent, configure the IngestionPipeline with a docstore and set a docstore_strategy. This persists document content hashes, enabling the pipeline to derive insert, update, or skip decisions during scheduled or webhook-triggered re-indexing.

LangChain supports idempotent ingestion through its index API combined with a RecordManager. This pairing enables explicit delete propagation strategies and cleanup modes, ensuring vector store synchronization when documents are edited or removed between runs.

Handling document deletions during RAG re-indexing requires an explicit delete propagation strategy. You must choose between snapshot-based cleanup for full corpus synchronization or incremental cleanup, because deletions are not automatically propagated to the vector store.

Test RAG ingestion idempotency using a twice-run regression gate. Run the pipeline once to populate the vector store, then run it again and verify the second run processes zero nodes and produces no changes to vector counts.

Yes, stable document identities are required to prevent duplicate vectors during upserts. Idempotent ingestion relies on a stable document content hash stored in a persisted docstore ledger to accurately determine if a document should be inserted, updated, or skipped.