hybrid-search-rag

Run PostgreSQL full-text search (ts_rank_cd with tsvector) and pgvector cosine similarity in parallel, then merge the ranked lists using Reciprocal Rank Fusion with k=60. RRF combines ranks without needing score calibration between the two retrievers.

Reciprocal Rank Fusion merges ranked lists by summing 1/(k + rank) for each document across lists. It works for RAG because BM25 scores and cosine similarities are not comparable, but ranks are, so documents appearing in both lists rank higher without score normalization.

Yes, PostgreSQL provides tsvector and tsquery with ts_rank_cd ranking, which implements BM25-style probabilistic ranking. Add a tsvector column with a GIN index and an update trigger to keep search vectors synchronized with content changes.

Use query expansion with a synonym taxonomy that maps terms like "CI/CD" to "deployment" and "e2e" to "testing". Expand query tokens before BM25 search so conceptual and abbreviated queries match canonical content.

Use a confidence score combining signals like BM25/vector overlap, top result quality, and margin between results. Return results directly above 0.80, return with refinement suggestions between 0.55-0.79, and ask a clarifying question below 0.55.

RRF ignores score magnitude, treating a BM25 score of 10.0 the same as 0.1 if ranks are equal. It is also sensitive to candidate list length, so fetching more results from one retriever biases the fusion toward it.