databricks-vector-search

Use the Databricks SDK create_index method with index_type DELTA_SYNC, specifying the source table, primary key, and an embedding source column. Databricks computes embeddings automatically using a model like databricks-gte-large-en.

Standard endpoints offer roughly 50-100ms query latency and hold up to 320M vectors, while Storage-Optimized endpoints hold over 1B vectors at about 7x lower cost with roughly 250ms latency. Storage-Optimized is the default choice unless you need sub-100ms latency.

Standard endpoints accept filters_json as a dictionary, while Storage-Optimized endpoints use filter_string with SQL-like syntax such as category = 'ai' AND status IN ('active', 'pending'). Pass the appropriate parameter to query_index.

Yes, use Delta Sync with self-managed embeddings by pointing to a Delta table containing an embedding vector column, or use a Direct Access index for full manual CRUD control over vectors via upsert and delete API calls.

Indexes with TRIGGERED pipeline type only update when you call sync_index manually. Switch to CONTINUOUS pipeline type for automatic syncing when the source Delta table changes.