skills.rest
Blackkadder
BlackkadderCOMMUNITY
Explore all skills by this author →
GitHub

databricks-ai-functions

Enrich, extract, classify, summarize, and parse data within Databricks SQL and PySpark pipelines.

Updated Mar 19, 2026
Search Tags:#rag#sql#pyspark#document-processing#databricks#ai-functions#ai-forecast
One-click install
npx skills add https://github.com/Blackkadder/databricks-apps-and-agents-workshop --skill databricks-ai-functions-blackkadder
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-ai-functions
Source: https://github.com/Blackkadder/databricks-apps-and-agents-workshop/tree/main/.claude/skills/databricks-ai-functions
Command: npx skills add https://github.com/Blackkadder/databricks-apps-and-agents-workshop --skill databricks-ai-functions-blackkadder

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Databricks AI Functions remove the operational burden of managing model endpoints by providing built-in functions that enrich, extract, classify, summarize, and parse data directly inside SQL and PySpark pipelines, enabling large-scale, production-ready AI integration with minimal infrastructure work.

Core Features & Use Cases

  • Task-specific functions for common needs: sentiment, classification, extraction, summarization, masking, translation, grammar fixes, generation, and similarity scoring to enrich table columns and speed up ETL tasks.
  • ai_parse_document and document pipelines for parsing PDFs and images into structured text, combined with extraction and RAG patterns to build invoice extraction, archival search, and document QA systems.
  • ai_query and ai_forecast for complex nested JSON outputs, multimodal inputs, custom model endpoints, and table-valued time series forecasting integrated with Delta tables and Lakehouse pipelines.

Quick Start

Parse a PDF with ai_parse_document, extract invoice fields with ai_extract or ai_query, and write the structured results to a Delta table.

Frequently Asked Questions about databricks-ai-functions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I use Databricks SQL to extract structured data from PDFs and images?▼

Databricks SQL extracts structured data from PDFs and images using ai_parse_document for parsing and ai_extract or ai_query for retrieving specific fields, writing structured results directly to Delta tables.

Can I perform sentiment analysis and text classification directly in PySpark pipelines?▼

PySpark pipelines perform sentiment analysis and text classification using built-in Databricks AI functions to enrich table columns without requiring external model endpoints or separate infrastructure.

Do I need to set up external model endpoints to use AI functions in Databricks?▼

AI functions in Databricks do not require external model endpoints. Built-in functions handle enrichment, extraction, and summarization directly within SQL and PySpark pipelines, removing operational infrastructure burden.

What are the DBR runtime requirements for parsing documents with Databricks AI functions?▼

Parsing documents with Databricks AI functions requires DBR 15.1 or higher, with advanced document parsing needing 17.1 or higher. These constraints ensure compatibility with Delta tables and Unity Catalog volumes.

How do I handle batch errors when applying AI functions to streaming data ingestion?▼

Batch errors during streaming data ingestion are handled using the failOnError parameter for batch-safe processing. This ensures Databricks AI functions perform table enrichment and document parsing without interrupting the pipeline.

What is the best way to build a RAG pipeline for invoice extraction in Databricks?▼

Building a RAG pipeline for invoice extraction in Databricks involves parsing PDFs with ai_parse_document, extracting fields with ai_extract or ai_query, and applying semantic similarity scoring to output structured Delta table records.

Related Skills

Explore More Data & Analytics Skills→
JingyiWngJingyiWng

databricks-agent-bricks

Create and orchestrate Databricks AI agent bricks for document Q&A, SQL exploration, and multi-agent supervision.

Community
Advanced
patrikboroschpatrikborosch
14

spark-consumption-cli

Run PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via Livy API sessions.

Community
Intermediate
RajuNaikRajuNaik
1

databricks-medallion-architecture

Design and implement Bronze/Silver/Gold medallion data architectures on Databricks.

Community
Advanced
teegin-gteegin-g

databricks-ai-functions

Run built-in AI inference in SQL and PySpark pipelines.

Community
Advanced