skills.rest
9vantage
9vantageCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

spark-consumption-cli

Analyze lakehouse data interactively using Fabric Lakehouse Livy API sessions and PySpark.

Updated Jul 14, 2026
Search Tags:#data analysis#pyspark#lakehouse#spark sql#delta lake#microsoft fabric#livy api
One-click install
npx skills add https://github.com/9vantage/skills-for-fabric-clone --skill spark-consumption-cli-9vantage
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-consumption-cli
Source: https://github.com/9vantage/skills-for-fabric-clone/tree/main/plugins/fabric-skills/skills/spark-consumption-cli
Command: npx skills add https://github.com/9vantage/skills-for-fabric-clone --skill spark-consumption-cli-9vantage

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running PySpark or Spark SQL analysis against Microsoft Fabric lakehouses requires managing Livy sessions, authentication tokens, and REST API calls manually, which is error-prone and slow without clear guidance. ## Core Features & Use Cases - Lakehouse Livy Session Management: Create, reuse, monitor, and clean up interactive Livy sessions via the Fabric REST API, with starter pool configuration for fast startup. - Interactive PySpark Analytics: Execute PySpark and Spark SQL statements for DataFrames, cross-lakehouse joins, Delta time-travel, and unstructured/JSON data exploration. - Workspace and Item Discovery: Dynamically resolve workspace and lakehouse IDs by name using JMESPath filtering instead of hardcoded identifiers. - Use Case: A data engineer needs to profile JSON files in a lakehouse and join them with Delta tables across two workspaces; this skill walks through session setup, statement execution, and cleanup via CLI. ## Quick Start Ask the AI to analyze a lakehouse table with PySpark by creating a Livy session and running a Spark DataFrame query against it.

Frequently Asked Questions about spark-consumption-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run PySpark against a Microsoft Fabric lakehouse from the CLI?▼

Create a Lakehouse Livy session via the Fabric REST API at /lakehouses/{id}/livyapi/.../sessions using az rest, wait for the session to reach idle state, then POST PySpark statements to the session's statements endpoint.

When should I use Livy sessions instead of the SQL endpoint for lakehouse queries?▼

Use the SQL endpoint via sqlcmd for simple queries like row counts, SELECT, filtering, and aggregation on Delta tables. Use Livy sessions only when you need PySpark DataFrames, unstructured or JSON data processing, or Spark-specific features like Delta time-travel.

What is the difference between Lakehouse Livy sessions and Notebook Spark sessions in Fabric?▼

Lakehouse Livy sessions are ad-hoc interactive sessions created through the public Livy API and managed directly. Notebook Spark sessions are created internally when a notebook runs via the Jobs API and are managed through the notebook lifecycle, not the Livy API.

How do I find a Fabric workspace ID or lakehouse ID by name?▼

List all workspaces with az rest against the Fabric API, then filter by display name using JMESPath. Repeat the pattern listing items of type Lakehouse within the workspace to resolve the lakehouse ID dynamically.

Why does my Fabric Livy session take long to start?▼

Session startup is slow when not using a starter pool. Configure spark.fabric.pool.name as Starter Pool in the session creation body, which typically makes sessions ready in 3-5 seconds instead of over a minute.

Related Skills

Explore More Data & Analytics Skills→
reason-machinesreason-machines
5

microsoft-fabric-unified-analytics

Builds a Microsoft Fabric analytics platform with Lakehouse, Dataflow Gen2, PySpark notebooks, and Power BI.

Official
Advanced
rritecrritec
3

fabriciq

Answer natural-language questions about Power BI report data by querying semantic models.

Community
Advanced
microsoftmicrosoft
952

sqldw-operations-cli

Analyze Fabric Data Warehouse performance and diagnostics via CLI.

Official
Advanced
patrikboroschpatrikborosch
14

spark-consumption-cli

Run PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via Livy API sessions.

Community
Intermediate