databricks-ai-functions

Name: databricks-ai-functions
Availability: InStock
Author: aaronachermann

Expose Databricks AI Functions to SQL and PySpark pipelines.

Updated Apr 18, 2026

Search Tags:#sql #forecast #pyspark #data-pipelines #databricks #document-parsing #ai-functions

Ready to run

npx skills add https://github.com/aaronachermann/PolentaEncoders --skill databricks-ai-functions-aaronachermann

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-ai-functions
Source: https://github.com/aaronachermann/PolentaEncoders/tree/main/.github/skills/databricks-ai-functions
Command: npx skills add https://github.com/aaronachermann/PolentaEncoders --skill databricks-ai-functions-aaronachermann

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Databricks AI Functions remove the need to manage external model endpoints by embedding AI capabilities directly into SQL and PySpark workflows, enabling teams to infuse intelligence into data pipelines with minimal setup.

Core Features & Use Cases

Task-specific AI functions cover sentiment analysis, classification, extraction, grammar correction, generation, masking, similarity, summarization, translation, and PDF/document parsing, plus ai_parse_document for document ingestion and ai_forecast for forecasting.
Seamless integration into existing data workloads: use in SELECT statements, withColumn operations, and Lakehouse pipelines to enrich data, extract structured fields, or summarize content.
Real-world scenario: classify support tickets, extract invoice fields, and generate a concise summary in a single processing step.

Quick Start

Run a simple query to classify text, extract key fields, and summarize results from a support_tickets table.

Frequently Asked Questions about databricks-ai-functions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I use AI functions directly in SQL pipelines for text classification and extraction?▼

You can embed AI functions directly in SQL pipelines using task-specific functions like ai_classify and ai_extract within SELECT statements to classify text and extract key fields without managing separate model endpoints. This allows real-time data enrichment inside existing data workloads.

Can I parse PDF documents and extract structured fields using Databricks SQL?▼

Yes, you can parse PDF documents in Databricks SQL using the ai_parse_document function. It enables document ingestion and extraction of structured fields directly inside your data pipelines, removing the need for external document parsing tools or separate model endpoints.

What is the best way to run sentiment analysis and forecasting in PySpark without external model endpoints?▼

The best way is to use built-in Databricks AI functions like ai_forecast for forecasting and task-specific sentiment functions within PySpark withColumn operations. This approach embeds AI capabilities directly into your workflows, eliminating the overhead of managing external model endpoints.

Does this approach to embedding AI functions in data pipelines require a specific Databricks environment?▼

Yes, embedding AI functions directly in SQL and PySpark pipelines requires a DBR 15.1 or higher environment. This specific Databricks Runtime version provides the native support needed to execute task-specific AI functions without provisioning separate model endpoints.

When should I use ai_query instead of built-in task-specific AI functions in Databricks?▼

You should use ai_query as a last-resort option when built-in task-specific functions cannot handle complex JSON parsing or multimodal needs. For standard tasks like classification, extraction, translation, and summarization, use dedicated functions like ai_classify, ai_extract, ai_translate, or ai_summarize.