chunk

Token-based chunking splits text based on token count for LLM input, while page-based chunking preserves document layout by dividing content into fixed page groups. The Skill supports both strategies to break down large PDFs into manageable pieces for LLM consumption.

The best way to chunk a Markdown file by token count is using a token-based splitting strategy. This approach divides the text based on token count, making the resulting chunks directly suitable for LLM input processing and analysis.

Yes, you can split a large PDF report into fixed page groups using the page-based chunking strategy. This method preserves the document layout, which is ideal for multimodal LLMs that process visual and textual information together.

PRD chunking hierarchically chunks documents like Product Requirements Documents while preserving their structure. This ensures that the logical flow and organizational hierarchy of the original document remain intact within each generated chunk.

Background execution processes large PDF files asynchronously during chunking, preventing interface blockage. This allows you to continue working while the Skill splits the document into token-based or page-based chunks in the background.

Yes, tiktoken is required to split text by token count for LLM input. The Skill uses this dependency alongside pymupdf to accurately measure and divide PDF and Markdown files into token-based chunks for processing.