databricks-vector-search

To create a vector search index for RAG, you provision a Databricks Vector Search endpoint and configure a Delta Sync or Direct Access index. This enables efficient semantic similarity matching and document retrieval for your pipeline.

Delta Sync indexes automatically synchronize vector embeddings directly from Delta tables, while Direct Access indexes require you to manually manage and ingest embeddings. Both facilitate semantic search but differ in embedding management overhead.

Yes, Databricks Vector Search supports both managed and self-managed embeddings. You can compute embeddings externally and ingest them into Direct Access indexes, or let Databricks compute them automatically using Delta Sync.

You can apply filters to semantic search queries by passing filtering conditions alongside your vector similarity search. This enables hybrid search capabilities, allowing you to restrict retrieval to specific metadata fields within your index.

Choose a storage-optimized endpoint for vector search when managing large-scale indexes where cost efficiency is prioritized over latency. Standard endpoints are better suited for high-performance, low-latency similarity matching workloads.

Yes, Databricks Vector Search supports hybrid search by combining semantic similarity matching with metadata filtering. This allows you to query indexes using both vector embeddings and traditional filters to refine RAG retrieval results.