indexing-pipeline

What problem does it solve?

Indexing Pipeline automates the end-to-end process of ingesting PDFs: loading content with multiple loaders, splitting into chunks with metadata, and preparing data for embeddings and graph-based retrieval.

Core Features & Use Cases

• Hybrid Loader + Splitter: combines PyMuPDF, Docling, and Vision OCR to extract text and tables; chunking with page awareness.
• Knowledge Graph Readiness: generates entities and relationships for GraphRAG and semantic search.
• Delta & Robustness: supports incremental indexing, deterministic IDs, and batch checkpoints for resuming failed runs.
• Use Case: index a large collection of PDFs to enable fast, context-enriched retrieval and knowledge graph construction.

Quick Start

Run the indexing pipeline on a PDF or directory to build the document index and knowledge graph.