databricks-vector-search

To create a Databricks vector index, deploy a Standard or Storage-Optimized endpoint, then build a Delta Sync or Direct Access index. Delta Sync indexes with managed embeddings automatically handle vector generation from your source Delta tables for semantic search workflows.

Delta Sync vector indexes automatically synchronize with source Delta tables and can use managed or self-managed embeddings. Direct Access indexes require you to supply pre-computed vectors, offering more control over the embedding pipeline for specialized semantic search use cases.

Yes, querying a Databricks vector index supports hybrid search combining keyword and semantic similarity, along with metadata filters. This enables precise retrieval for RAG pipelines by narrowing vector search results based on specific document attributes.

You need an embedding model for managed Delta Sync and Direct Access indexes to generate vector embeddings from text. Self-managed Delta Sync indexes allow you to supply pre-computed embeddings, bypassing the need for an internal model.

Set up a Databricks Vector Search endpoint by deploying either a Standard or Storage-Optimized instance via the manage_vs_endpoint tool. Standard endpoints suit general workloads, while Storage-Optimized endpoints handle larger vector indexes for scalable document similarity.

Use a Storage-Optimized endpoint for vector search when managing large-scale document similarity matching or extensive RAG pipelines. It prioritizes storage efficiency over compute speed, making it suitable for batch workloads with massive Delta Sync or Direct Access indexes.