skills.rest
microsoft
microsoftOFFICIAL
Explore all skills by this author →
Website𝕏@OpenAtMicrosoftGitHub

spark-cli

Manages Fabric Spark notebooks, Livy sessions, failure diagnostics, and Materialized Lake View lifecycles via REST APIs.

1.1k|301|Updated Feb 17, 2026
Search Tags:#spark#Diagnostics#REST api#livy#materialized lake view#Notebook#Microsoft Fabric
One-click install
npx skills add https://github.com/microsoft/skills-for-fabric --skill spark-cli
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-cli
Source: https://github.com/microsoft/skills-for-fabric/tree/main/plugins/fabric-skills/skills/spark-cli
Command: npx skills add https://github.com/microsoft/skills-for-fabric --skill spark-cli

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Working with Microsoft Fabric Spark requires juggling notebook authoring, Livy session management, job failure triage, and Materialized Lake View (MLV) operations across many REST endpoints, and mistakes like wrong payload shapes or missed terminal writes produce silent failures.

Core Features & Use Cases

  • Notebook authoring and execution: Write PySpark, Scala, SparkR, %%sql, and %%configure cells, save them via updateDefinition, and run named notebooks through the Jobs API.
  • Interactive Livy sessions: Create and reuse Lakehouse Livy sessions for ad-hoc PySpark analysis, cross-lakehouse queries, and Delta time-travel.
  • Spark failure diagnostics: Triage failed, slow, or OOM notebook and pipeline runs using the Spark Advisor API, monitoring APIs, and log retrieval.
  • Materialized Lake View lifecycle: Author MLV definitions, review queries for incremental-refresh readiness, schedule refreshes, trigger on-demand runs, and diagnose refresh failures.
  • Use Case: A user asks why their nightly notebook failed; the skill resolves the workspace, queries the Spark Advisor, identifies an OutOfMemoryError with data skew, and reports the root cause with a Notebook Snapshot link.

Quick Start

Ask the assistant to diagnose why your Fabric notebook run failed, or to schedule a nightly refresh for a Materialized Lake View in your lakehouse.

Frequently Asked Questions about spark-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Fabric notebook by name using the REST API?▼

Resolve the workspace and notebook IDs by listing items and filtering by name, then trigger the run via the Jobs API with POST /items/{notebookId}/jobs/instances?jobType=RunNotebook. Capture the job instance ID immediately and poll status instead of retrying the POST.

How do I schedule a Materialized Lake View refresh in Microsoft Fabric?▼

Use POST /workspaces/{id}/lakehouses/{id}/jobs/refreshMaterializedLakeViews/schedules with a Daily, Weekly, Monthly, or Cron configuration including startDateTime, endDateTime, and localTimeZoneId. A lakehouse supports at most 20 schedules and one active schedule per MLV lineage.

What is the difference between a Lakehouse Livy session and a notebook Spark session?▼

Lakehouse Livy sessions are created through the public Livy API for ad-hoc interactive PySpark code. Notebook Spark sessions are created internally when a notebook runs via the Jobs API and are not managed through the Livy API.

Why did my Fabric Spark notebook fail with an OutOfMemoryError?▼

Query the Spark Advisor API first, since it pre-computes task errors, skew, and memory pressure findings without log parsing. Confirm with stage metrics such as a max-to-median task ratio above 3x or disk spill before recommending executor memory or partitioning changes.

Can I list Materialized Lake Views through the Fabric REST API?▼

No, the REST discovery endpoint returns 404. Run SHOW MATERIALIZED LAKE VIEWS IN <schema> through a Lakehouse Livy session or a notebook cell to discover MLVs instead.

When should KQL materialized view requests not use this skill?▼

KQL materialized views in an Eventhouse belong to the eventhouse-cli skill, not spark-cli. If that skill is unavailable, the request cannot be completed and no Fabric API calls or artifacts should be created.

Related Skills

Explore More Data & Analytics Skills→
rritecrritec
3

spark-consumption-cli

Analyze Microsoft Fabric lakehouse data with interactive PySpark sessions and Spark SQL.

Community
Advanced
patrikboroschpatrikborosch
14

e2e-medallion-architecture

Automate Bronze/Silver/Gold medallion lakehouse pipelines in Microsoft Fabric.

Community
Advanced
PatrickGallucciPatrickGallucci
16

fabric-spark-perf-remediate

Diagnose and remediate Fabric Spark performance problems across workloads.

Community
Advanced
microsoftmicrosoft
952

sqldw-consumption-cli

Query read-only T-SQL against Fabric Data Warehouse, Lakehouse, and Mirrored Databases via CLI.

Official
Intermediate