skills.rest
TRRaveendra
TRRaveendraCOMMUNITY
Explore all skills by this author →
GitHub

sdlc-agent-07-pyspark-development

Generates production PySpark notebooks from data models and DDL on Databricks.

1|2|Updated Aug 10, 2026
Search Tags:#code generation#etl#sdlc#pyspark#databricks#delta lake#foundation model api
One-click install
npx skills add https://github.com/TRRaveendra/AI-Agents-Databricks --skill sdlc-agent-07-pyspark-development-trraveendra
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sdlc-agent-07-pyspark-development
Source: https://github.com/TRRaveendra/AI-Agents-Databricks/tree/main/ai-agent-skills/databricks-sdlc-agent-skills/07-pyspark-development-agent
Command: npx skills add https://github.com/TRRaveendra/AI-Agents-Databricks --skill sdlc-agent-07-pyspark-development-trraveendra

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-sdk, mlflow, and includes references (resource) components.

What problem does it solve? It automates stage 7 of a 15-agent Databricks SDLC pipeline by converting upstream data models, DDL, and mapping documents into validated, production-ready PySpark notebooks without manual coding. ## Core Features & Use Cases - Metadata-driven code generation: Calls the Databricks Foundation Model API to produce incremental PySpark with Delta MERGE/SCD logic, error handling, logging, and audit trails. - Contract-validated output: Returns JSON matching a strict output contract, writes artifacts to Unity Catalog Volumes, and logs runs to state.agent_runs for auditability. - Smoke-test execution: Runs the generated notebook on a scratch job cluster or serverless compute, with one repair retry on failure before routing to a human. - Use Case: A data engineering team receives a data model from Agent 6 and needs incremental Delta Lake notebooks following medallion conventions; this agent generates, smoke-tests, and hands off the artifact to the SQL Agent automatically. ## Quick Start Ask the agent to generate a PySpark notebook from the data model and DDL artifacts for the current ticket and write the result to the artifacts volume.

Frequently Asked Questions about sdlc-agent-07-pyspark-development

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate PySpark notebooks automatically on Databricks?▼

Call the Databricks Foundation Model API (Model Serving chat completions) with the data model, DDL, and mapping document as context, requiring code-only output. Then smoke-test the generated notebook on a scratch job cluster or serverless compute before marking it complete.

Can Genie Code be called via API from an orchestrator?▼

No, Genie Code is UI-only with no public REST or SDK endpoint for headless invocation. Use the Foundation Model API or a deployed Mosaic AI Agent endpoint instead for code generation steps in automated pipelines.

What inputs does the PySpark Development Agent need?▼

It requires the Data Model and DDL artifact from Agent 6 and the Mapping Document from Agent 5, read from /Volumes/{catalog}/state/artifacts/{ticket}/. If required context is missing, it returns status needs_clarification rather than inventing metadata.

What happens when generated PySpark code fails validation?▼

The agent performs one repair retry with the validation error or stack trace appended to the prompt context. If the retry also fails, it routes the ticket to state.needs_human for manual review instead of looping indefinitely.

Does this agent use the Genie Agents Conversation API?▼

No, this agent has no natural-language data-querying step, so it does not need the Genie Agents API. That API is reserved for agents 3, 4, 8, and 15 which answer ad-hoc business or profiling questions.

Related Skills

Explore More Software Engineering Skills→
RDEWAIRDEWAI
5

create-stm

Generate standardized source-to-target mapping Excel workbooks for medallion data pipelines.

Official
Advanced
linus-mcmanameylinus-mcmanamey

schema-reference

Validate PySpark ETL schemas against DuckDB metadata and data dictionaries.

Community
Intermediate
ThomazRossitoThomazRossito
31

databricks-spark-declarative-pipelines

Create and configure Databricks Lakeflow Spark Declarative Pipelines with serverless compute.

Community
Advanced
JingyiWngJingyiWng

databricks-agent-bricks

Create and orchestrate Databricks AI agent bricks for document Q&A, SQL exploration, and multi-agent supervision.

Community
Advanced