databricks-vector-search

To set up vector search in Databricks for RAG, create a Standard or Storage-Optimized endpoint, then configure a Delta Sync or Direct Access index. This enables fast, scalable similarity matching against Delta Lake data to retrieve relevant documents for AI assistants.

Delta Sync indexes automatically synchronize changes from Delta Lake tables, while Direct Access indexes require manual upserts and management. Delta Sync supports managed or self-managed embedding workflows, whereas Direct Access offers more granular control over vector updates.

Yes, Databricks vector search supports hybrid search and filtering. You can perform queries against your index to fetch top results while applying filters to narrow the scope, enabling precise semantic search and similarity matching across your Delta Lake data.

Yes, vector search requires a dedicated endpoint. You can choose between Standard endpoints for general workloads or Storage-Optimized endpoints to handle large-scale vector retrieval, ensuring scalable and fast performance for RAG applications.

Databricks handles embeddings through Delta Sync indexes with managed or self-managed embedding options, or via Direct Access indexes where you supply the vectors. This flexibility allows you to integrate embeddings seamlessly for semantic search and similarity matching.

Direct Access indexes do not automatically synchronize with Delta Lake changes, requiring manual upserts and scans to maintain data freshness. This approach shifts the burden of embedding handling and index updates to the user, unlike the automated Delta Sync alternative.