wiki-chunk

To chunk transcripts for RAG pipelines, you segment pre-transcription text into topic-based chunks with timestamps and key entities. This process outputs structured JSON files suitable for embedding and indexing to enable fast search and relevance ranking.

Yes, you can use Gemini for topic segmentation of transcripts. The pipeline leverages Gemini to generate coherent topic boundaries and context, splitting transcripts into semantic segments with timestamps and topic types for structured retrieval.

The best way to structure transcript chunks for embedding is semantic topic segmentation. This approach splits transcripts into topic-based chunks with timestamps, topic types, and key entities, emitting structured per-episode JSON arrays for efficient downstream embedding.

Yes, you need a GEMINI_API_KEY set in your environment to generate topic-based JSON chunks. The pipeline requires this API key, along with dependencies like google-genai and pyyaml, and a wiki.yaml configuration file to process transcripts.

Semantic chunking improves search relevance by splitting transcripts into coherent topic segments with context. This allows retrieval-augmented generation pipelines to surface relevant topic segments quickly, enabling fast search and structured retrieval over large corpora.

The chunking pipeline outputs per-episode JSON files containing a structured array of chunks. These files are emitted into a chunks directory and are formatted specifically for downstream embedding and indexing in retrieval-augmented workflows.