databricks-ai-functions

You can embed AI functions directly in SQL pipelines using task-specific functions like ai_classify and ai_extract within SELECT statements to classify text and extract key fields without managing separate model endpoints. This allows real-time data enrichment inside existing data workloads.

Yes, you can parse PDF documents in Databricks SQL using the ai_parse_document function. It enables document ingestion and extraction of structured fields directly inside your data pipelines, removing the need for external document parsing tools or separate model endpoints.

The best way is to use built-in Databricks AI functions like ai_forecast for forecasting and task-specific sentiment functions within PySpark withColumn operations. This approach embeds AI capabilities directly into your workflows, eliminating the overhead of managing external model endpoints.

Yes, embedding AI functions directly in SQL and PySpark pipelines requires a DBR 15.1 or higher environment. This specific Databricks Runtime version provides the native support needed to execute task-specific AI functions without provisioning separate model endpoints.

You should use ai_query as a last-resort option when built-in task-specific functions cannot handle complex JSON parsing or multimodal needs. For standard tasks like classification, extraction, translation, and summarization, use dedicated functions like ai_classify, ai_extract, ai_translate, or ai_summarize.