skills.rest
9vantage
9vantageCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

spark-operations-cli

Diagnose failed Spark jobs, Livy sessions, and performance bottlenecks in Microsoft Fabric via read-only CLI triage.

Updated Jul 14, 2026
Search Tags:#performance#troubleshooting#pipeline#diagnostics#spark#microsoft fabric#livy
One-click install
npx skills add https://github.com/9vantage/skills-for-fabric-clone --skill spark-operations-cli-9vantage
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-operations-cli
Source: https://github.com/9vantage/skills-for-fabric-clone/tree/main/plugins/fabric-operations/skills/spark-operations-cli
Command: npx skills add https://github.com/9vantage/skills-for-fabric-clone --skill spark-operations-cli-9vantage

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? When a Spark notebook, job, or pipeline fails in Microsoft Fabric, finding the root cause requires digging through scattered logs, session states, and monitoring APIs. This Skill automates that triage with a systematic, read-only diagnostic workflow. ## Core Features & Use Cases - Failure Triage: Classify failures (driver/executor OOM, shuffle fetch failures, executor loss, library errors) using the Spark Advisor API, driver/executor logs, and job instance history. - Performance Diagnostics: Detect data skew, disk spill, GC pressure, heavy shuffles, and low core efficiency using stage metrics and the Resource Usage API with defined severity thresholds. - Pipeline Run Diagnosis: Query queryActivityRuns to diagnose every Spark activity in a failed pipeline run, including ForEach iterations, retries, and nested pipelines, with cell-level traceback extraction. - Offline Deep-Dive: Copy event logs to OneLake via the JobInsight API and start a local Spark History Server when online monitoring data is truncated or expired. - Use Case: A user asks "Why did my notebook ETL_Daily fail in workspace Production?" The Skill resolves the workspace and notebook, finds the failed Livy session, queries Spark Advisor, confirms data skew in stage metrics, and returns a severity-ordered report with fixes and a Notebook Snapshot link. ## Quick Start Ask the assistant to diagnose why your Fabric notebook or Spark job failed, providing the workspace name and item name or a pipeline run ID.

Frequently Asked Questions about spark-operations-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose a failed Spark notebook in Microsoft Fabric?▼

Provide the workspace and notebook name, and the Skill resolves the IDs, lists recent Livy sessions, and auto-selects the failed one. It then queries the Spark Advisor API, driver stderr logs, and job instance history to classify the root cause and suggest fixes.

How to find the root cause of a failed Fabric pipeline run?▼

The Skill calls the queryActivityRuns API for the pipeline run ID to retrieve every activity's status, session ID, and full Python traceback with cell and line numbers. It then runs Spark diagnostics on each failed Notebook or Spark Job activity and compiles a combined report.

What is the difference between spark-operations-cli and spark-authoring-cli?▼

spark-operations-cli is strictly for read-only triage and diagnosis of existing jobs and sessions. For creating notebooks, running new jobs, or Spark development work, use spark-authoring-cli; for interactive PySpark analysis, use spark-consumption-cli.

Why do Spark Monitoring API calls return 404 for older sessions?▼

Spark Monitoring data such as logs, stages, and advisor findings expires within minutes to hours after a session ends. The Skill falls back to queryActivityRuns, job instance failureReason, and the Notebook Snapshot URL, which have longer retention.

Can I analyze the full Spark UI for a completed Fabric job?▼

Yes, through the Tier 2 offline workflow. The Skill copies event logs to OneLake using the JobInsight LogUtils.copyEventLog Scala API, downloads them locally, and starts a local Spark History Server at localhost:18080 for DAG and SQL plan visualization.

How do I detect data skew or memory problems in a Fabric Spark job?▼

The Skill checks the Spark Advisor API first, then analyzes stage and executor metrics against thresholds: max/median task duration above 3x indicates skew, diskBytesSpilled above zero indicates memory pressure, and GC time above 20 percent of run time indicates heap pressure.

Related Skills

Explore More Data & Analytics Skills→
weiflycc-cmdweiflycc-cmd

spark-operations-cli

Diagnose failed Spark jobs and Livy sessions in Microsoft Fabric via read-only CLI.

Community
Advanced
SanketAdlakSanketAdlak

spark-optimization

Optimizes Apache Spark jobs through partitioning, caching, shuffle tuning, and memory configuration.

Community
Advanced
PatrickGallucciPatrickGallucci
16

fabric-spark-perf-remediate

Diagnose and remediate Fabric Spark performance problems across workloads.

Community
Advanced
microsoftmicrosoft
952

spark-operations-cli

Diagnose Spark job failures and Livy session health using read-only REST APIs.

Official
Advanced