skills.rest
ice-droid-99
ice-droid-99COMMUNITY
Explore all skills by this author →
GitHub

databricks-ai-functions

Enrich Databricks SQL and PySpark data with built-in AI functions.

Updated May 20, 2026
Search Tags:#information extraction#document parsing#rag pipeline#pii redaction#databricks sql#ai functions#time series forecasting
One-click install
npx skills add https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks- --skill databricks-ai-functions-ice-droid-99
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-ai-functions
Source: https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks-/tree/main/.gemini/skills/databricks-ai-functions
Command: npx skills add https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks- --skill databricks-ai-functions-ice-droid-99

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill solves the problem of building AI-powered analysis, extraction, and document understanding directly inside Databricks SQL and PySpark without manually provisioning and managing model endpoints.

Core Features & Use Cases

  • Task-specific AI functions for pipelines: Use built-in functions such as ai_classify, ai_extract, ai_summarize, and ai_mask to enrich columns with sentiment, entities, summaries, and PII redaction.
  • Document parsing and enrichment: Use ai_parse_document to convert PDFs and other document binaries into structured page/element text for downstream processing.
  • Fallback to general reasoning and structured outputs: Use ai_query for complex nested JSON extraction, multimodal inputs, and scenarios not covered by task-specific functions.
  • Forecasting as a table-valued function: Use ai_forecast to generate confidence-bounded time series predictions when the environment supports it.
  • End-to-end document processing & RAG patterns: Build batch pipelines (parse → classify → extract → validate → sidecar errors) and RAG workflows (parse → chunk → index → query) with guardrails for error handling.

Quick Start

Use this skill by running a Databricks SQL query that parses a document column with ai_parse_document, then extracts structured fields from the parsed text with ai_extract and summarizes results with ai_summarize.

Frequently Asked Questions about databricks-ai-functions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured entities and summarize text directly in Databricks SQL?▼

To extract structured entities and summarize text in Databricks SQL, use built-in functions like ai_extract and ai_summarize. These task-specific functions enrich columns with classifications and summaries without manually provisioning model endpoints.

Can I parse PDFs and mask PII data in a PySpark pipeline without managing endpoints?▼

Yes, you can parse PDFs and mask PII data in PySpark without managing endpoints. Use ai_parse_document to convert document binaries into structured text, then apply ai_mask to automatically redact sensitive information in your pipeline.

When should I use ai_query instead of task-specific AI functions for text classification?▼

You should use ai_query instead of task-specific functions for text classification when you need complex nested JSON extraction, multimodal inputs, or when scenarios are not covered by functions like ai_classify. It serves as a fallback for advanced reasoning.

Does Databricks support time series forecasting as a table-valued function?▼

Databricks supports time series forecasting as a table-valued function through ai_forecast. It generates confidence-bounded predictions and returns time series table outputs when the environment supports the required DBR or warehouse runtime.

What is the best way to handle errors during batch inference and document parsing?▼

The best way to handle errors during batch inference and document parsing is using the failOnError pattern. This safe error handling approach captures sidecar errors while processing parsed VARIANT outputs for downstream transformations.

Do I need to define a schema for structured extraction with AI functions?▼

Yes, you need to define a schema or responseFormat for structured extraction with AI functions. Structured extraction requires schemas to parse VARIANT outputs correctly and ensure proper handling for downstream transformations.

Related Skills

Explore More Data & Analytics Skills→
Zack2626-okZack2626-ok

databricks-ai-functions

Invoke Databricks AI Functions for sentiment analysis, classification, and summarization in SQL and PySpark pipelines.

Community
Advanced
mattwoodcomattwoodco
3

ai-rag-vectors

Create a pgvector embedding layer for semantic search with recursive text chunking.

Community
Advanced
teegin-gteegin-g

databricks-ai-functions

Run built-in AI inference in SQL and PySpark pipelines.

Community
Advanced
PaldomPaldom
11

databricks-ai-functions

Run Databricks built-in AI Functions directly from SQL and PySpark.

Community
Advanced