skills.rest
Mercurium-Analytics
Mercurium-AnalyticsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

bm25-tokenizer-guide

Guide per-field tokenizer selection for BM25 indexes with a decision tree.

Updated Apr 22, 2026
Search Tags:#indexing#autocomplete#postgres#text-search#bm25#tokenizer#pg_search-vector
One-click install
npx skills add https://github.com/Mercurium-Analytics/pg-search-vector --skill bm25-tokenizer-guide
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bm25-tokenizer-guide
Source: https://github.com/Mercurium-Analytics/pg-search-vector/tree/main/skills/bm25-tokenizer-guide
Command: npx skills add https://github.com/Mercurium-Analytics/pg-search-vector --skill bm25-tokenizer-guide

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Tokenizer selection for BM25 indexes is critical to balance search accuracy, index size, and performance. This guide helps you decide per-field tokenizers to optimize autocomplete for names and efficient full-text search for prose or code.

Core Features & Use Cases

  • Provides a decision-tree approach for choosing among default, ngram prefix (autocomplete), ngram substring, and code tokenizers.
  • Maps per-field strategies (names, descriptions, filenames) to practical index configurations and explains trade-offs in index size and hit quality.
  • Use Case: design a hybrid search in a catalog where product names need fast suggestions while descriptions deliver robust phrase matching.

Quick Start

Use the decision tree to configure per-field tokenizers and build your BM25 index accordingly.

Frequently Asked Questions about bm25-tokenizer-guide

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose a BM25 tokenizer for PostgreSQL text search?▼

Choosing a BM25 tokenizer for PostgreSQL text search requires balancing search accuracy against index size using a per-field decision tree that evaluates default, ngram, and code tokenization strategies based on your specific workload.

What is the best way to configure BM25 tokenizers for autocomplete versus full-text search?▼

Configuring BM25 tokenizers for autocomplete versus full-text search involves mapping ngram prefix tokenizers to name fields for fast suggestions, while applying default or code tokenizers to description fields for robust phrase matching.

When should I use an ngram substring tokenizer in a pg_search BM25 index?▼

You should use an ngram substring tokenizer in a pg_search BM25 index when your text-search workload requires matching characters within larger words, accepting a significantly larger index size in exchange for broader hit quality.

Does the BM25 code tokenizer affect index size in Postgres?▼

The BM25 code tokenizer affects Postgres index size by altering how filenames and code snippets are split into searchable terms, requiring careful per-field configuration to prevent unnecessary index bloat during tokenization.

Can I apply different BM25 tokenizers to individual table fields in Postgres?▼

You can apply different BM25 tokenizers to individual table fields in Postgres by implementing a per-field decision tree that isolates ngram prefix autocomplete for names and default tokenization for prose descriptions.

Related Skills

Explore More Software Engineering Skills→
Mercurium-AnalyticsMercurium-Analytics

django-pgsearch-patterns

Integrate Django with pg_search and pgvector for hybrid BM25 and vector search.

Official
Advanced
JNZader-VaultJNZader-Vault

search-concepts

Explains search engine concepts including indexing, inverted indexes, and relevance scoring.

Official
Intermediate
mattstyles333mattstyles333

PostgreSQL

Optimize PostgreSQL queries, schemas, indexing, and database performance.

Community
Advanced
48Nauts-Operator48Nauts-Operator
3

hybrid-search-implementation

Implement hybrid search combining vector similarity with BM25 ranking.

Community
Advanced