skills.rest
TRRaveendra
TRRaveendraCOMMUNITY
Explore all skills by this author →
GitHub

sdlc-agent-04-metadata-discovery

Discovers Unity Catalog metadata and builds data dictionaries for Databricks SDLC pipelines.

1|2|Updated Aug 10, 2026
Search Tags:#data dictionary#lineage#databricks#unity catalog#metadata discovery#sdlc pipeline#genie api
One-click install
npx skills add https://github.com/TRRaveendra/AI-Agents-Databricks --skill sdlc-agent-04-metadata-discovery-trraveendra
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sdlc-agent-04-metadata-discovery
Source: https://github.com/TRRaveendra/AI-Agents-Databricks/tree/main/ai-agent-skills/databricks-sdlc-agent-skills/04-metadata-discovery-agent
Command: npx skills add https://github.com/TRRaveendra/AI-Agents-Databricks --skill sdlc-agent-04-metadata-discovery-trraveendra

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-sdk, and includes references (resource) components.

What problem does it solve? Data engineering teams building ETL pipelines on Databricks often lack a governed, automated way to discover existing schemas, lineage, and data profiles before development begins, leading to duplicated work and fabricated assumptions about source data. ## Core Features & Use Cases - Unity Catalog Metadata Discovery: Reads information_schema, table and column comments, tags, and lineage from system.access.table_lineage to build a grounded data dictionary. - Governed Data Profiling: Uses the Genie Agents Conversation API for ad-hoc profiling questions (row counts, null rates, distinct values) instead of raw SQL, keeping results governed and explainable. - Contract-Validated Output: Produces a metadata_discovery.json artifact validated against a strict JSON output contract, written to /Volumes/{catalog}/state/artifacts/{ticket}/ for the next pipeline agent. - Use Case: As stage 4 of a 15-agent SDLC pipeline, it consumes a Business Understanding Document from the upstream agent and hands a validated Metadata Discovery Package to the Mapping Document Agent. ## Quick Start Ask the agent to read the Business Understanding Document for a ticket and produce the Metadata Discovery Package artifact for the next pipeline stage.

Frequently Asked Questions about sdlc-agent-04-metadata-discovery

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I discover table metadata in Databricks Unity Catalog?▼

Query information_schema, system.information_schema.columns, table and column comments, tags, and lineage from system.access.table_lineage. This agent automates that discovery and compiles the results into a governed data dictionary artifact.

How to profile Databricks tables without running raw SQL?▼

Use the Genie Agents Conversation API to ask natural-language profiling questions such as row counts, null rates, and distinct values against a Genie Space scoped to the relevant catalog and schema. This keeps results governed and explainable.

Can Genie Code be called headlessly from a Databricks job?▼

No, Genie Code is UI-only with no public REST or SDK endpoint for headless invocation. Use the Foundation Model API on Model Serving or a Mosaic AI Agent endpoint for code and SQL generation instead.

What happens when required metadata is not found in Unity Catalog?▼

The agent returns status needs_clarification with specific questions rather than fabricating schema. On failed contract validation it attempts one repair retry, then routes to state.needs_human for review.

What output format does the metadata discovery agent produce?▼

It returns JSON matching a strict output contract with fields for status, artifact_path, summary, citations, tool_calls, tokens, and duration. The artifact is written to /Volumes/{catalog}/state/artifacts/{ticket}/04_metadata-discovery-agent/ and logged in state.artifacts and state.agent_runs.

Related Skills

Explore More Data & Analytics Skills→
fsollafsolla

create-business

Orchestrate a multi-phase business validation workflow generating project documentation.

Community
Advanced
dbt-labsdbt-labs
699

using-dbt-for-analytics-engineering

Builds, tests, and validates dbt models using ref(), source(), and dbt show.

Official
Advanced
thewoolleymanthewoolleyman

AgentDB Vector Search

Perform semantic vector search over documents using AgentDB with HNSW indexing.

Community
Advanced
databricks-solutionsdatabricks-solutions
5

adhoc-exploration-notebooks

Generate dual-format exploration notebooks for Databricks and Jupyter with helper functions.

Official
Intermediate