spark-operations-cli

Diagnose failed Spark jobs and unhealthy Livy sessions using read-only CLI triage that queries Spark Monitoring APIs and Livy logs to identify root causes like OOM errors and capacity throttling without direct cluster access.

Yes, Azure authentication and Fabric REST API access via az rest are required to automatically resolve workspace and item IDs, query Spark Monitoring APIs, and generate structured diagnostic reports with severity classifications.

Trace pipeline run failures across ForEach iterations and nested pipelines by querying job instances and Livy logs via REST APIs, extracting exact cell-level traceback to pinpoint the failing notebook or Spark Job Definition.

Detect data skew, shuffle spill, and GC pressure by analyzing stage metrics and resource usage APIs, identifying underutilized capacity and performance bottlenecks across Fabric workspaces to generate prioritized fixes.

Monitor idle or zombie Livy sessions, capacity pressure, and stuck startup states across workspaces by querying session health APIs, detecting sessions that waste capacity resources and require cleanup.

Notebook ETL failures with capacity throttling occur when Fabric capacity limits are exceeded; query resource usage APIs and Spark Advisor to classify severity and identify whether throttling or data skew is the root cause.