bm25-tokenizer-guide

Guide per-field tokenizer selection for BM25 indexes with a decision tree.

Updated Apr 22, 2026

Search Tags:#indexing #autocomplete #postgres #text-search #bm25 #tokenizer #pg_search-vector

Ready to run

npx skills add https://github.com/Mercurium-Analytics/pg-search-vector --skill bm25-tokenizer-guide

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: bm25-tokenizer-guide
Source: https://github.com/Mercurium-Analytics/pg-search-vector/tree/main/skills/bm25-tokenizer-guide
Command: npx skills add https://github.com/Mercurium-Analytics/pg-search-vector --skill bm25-tokenizer-guide

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Tokenizer selection for BM25 indexes is critical to balance search accuracy, index size, and performance. This guide helps you decide per-field tokenizers to optimize autocomplete for names and efficient full-text search for prose or code.

Core Features & Use Cases

Provides a decision-tree approach for choosing among default, ngram prefix (autocomplete), ngram substring, and code tokenizers.
Maps per-field strategies (names, descriptions, filenames) to practical index configurations and explains trade-offs in index size and hit quality.
Use Case: design a hybrid search in a catalog where product names need fast suggestions while descriptions deliver robust phrase matching.

Quick Start

Use the decision tree to configure per-field tokenizers and build your BM25 index accordingly.

Frequently Asked Questions about bm25-tokenizer-guide

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I choose a BM25 tokenizer for PostgreSQL text search?▼

Choosing a BM25 tokenizer for PostgreSQL text search requires balancing search accuracy against index size using a per-field decision tree that evaluates default, ngram, and code tokenization strategies based on your specific workload.

What is the best way to configure BM25 tokenizers for autocomplete versus full-text search?▼

Configuring BM25 tokenizers for autocomplete versus full-text search involves mapping ngram prefix tokenizers to name fields for fast suggestions, while applying default or code tokenizers to description fields for robust phrase matching.

When should I use an ngram substring tokenizer in a pg_search BM25 index?▼

You should use an ngram substring tokenizer in a pg_search BM25 index when your text-search workload requires matching characters within larger words, accepting a significantly larger index size in exchange for broader hit quality.

Does the BM25 code tokenizer affect index size in Postgres?▼

The BM25 code tokenizer affects Postgres index size by altering how filenames and code snippets are split into searchable terms, requiring careful per-field configuration to prevent unnecessary index bloat during tokenization.

Can I apply different BM25 tokenizers to individual table fields in Postgres?▼

You can apply different BM25 tokenizers to individual table fields in Postgres by implementing a per-field decision tree that isolates ngram prefix autocomplete for names and default tokenization for prose descriptions.