skills.rest
microsoft
microsoftOFFICIAL
Explore all skills by this author →
Website𝕏@OpenAtMicrosoftGitHub

spark-consumption-cli

Run interactive PySpark and Spark SQL queries on Fabric Lakehouse Livy sessions.

952|267|Updated Feb 17, 2026
Search Tags:#pyspark#lakehouse#data-exploration#spark-sql#livy#cross-lakehouse#delta-time-travel
One-click install
npx skills add https://github.com/microsoft/skills-for-fabric --skill spark-consumption-cli
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-consumption-cli
Source: https://github.com/microsoft/skills-for-fabric/tree/main/plugins/fabric-skills/skills/spark-consumption-cli
Command: npx skills add https://github.com/microsoft/skills-for-fabric --skill spark-consumption-cli

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Analyze lakehouse data interactively using Fabric Lakehouse Livy API sessions and PySpark/Spark SQL for advanced analytics, DataFrames, cross-lakehouse joins, Delta time-travel, and unstructured/JSON data. Use when the user explicitly asks for PySpark, Spark DataFrames, Livy sessions, or Python-based analysis — NOT for simple SQL queries.

Core Features & Use Cases

  • Interactive Lakehouse analysis with PySpark and Spark SQL across multiple lakehouses
  • Manage Lakehouse Livy sessions (idle/session states) and execute statements
  • Cross-lakehouse joins, Delta time-travel, and handling unstructured/JSON data
  • Use cases: ad-hoc analytics, data discovery, and Python-based analytics workflows

Quick Start

Launch or reuse a Lakehouse Livy session and run Spark SQL or PySpark queries to explore lakehouse data across workspaces.

Frequently Asked Questions about spark-consumption-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run PySpark queries across multiple lakehouses in Fabric?▼

To run PySpark queries across multiple lakehouses, you need an active Fabric Lakehouse Livy session. This allows you to execute cross-lakehouse joins and Spark SQL statements interactively for ad-hoc analytics and data discovery.

Can I query previous versions of Delta tables using Spark SQL in a lakehouse?▼

Yes, you can query previous versions of Delta tables using Delta time-travel within your Spark SQL or PySpark environment. This requires an active Lakehouse Livy session to execute the analytical statements against your lakehouse data.

How do I analyze unstructured JSON data in a Fabric lakehouse?▼

You analyze unstructured JSON data by launching a Lakehouse Livy session and applying PySpark DataFrames. This interactive environment enables you to parse, transform, and explore raw JSON records alongside your structured lakehouse data.

What is the difference between using PySpark Livy sessions and simple SQL for lakehouse exploration?▼

PySpark Livy sessions are designed for advanced analytics, Python-based workflows, and DataFrame manipulations. You should use simple SQL queries for basic exploration, but switch to PySpark when handling cross-lakehouse joins or unstructured JSON data.

Do I need an active Lakehouse Livy session to execute cross-lakehouse queries?▼

Yes, an active Lakehouse Livy session is required to execute cross-lakehouse queries. You must also have a PySpark or Spark SQL environment configured to run these interactive analytical statements successfully.

Why use PySpark DataFrames instead of standard SQL for Fabric lakehouse data discovery?▼

PySpark DataFrames provide advanced analytical capabilities for handling unstructured JSON data and complex cross-lakehouse joins. Standard SQL is suitable for simple queries, but Python-based workflows are necessary for advanced data exploration.

Related Skills

Explore More Data & Analytics Skills→
patrikboroschpatrikborosch
14

spark-consumption-cli

Run PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via Livy API sessions.

Community
Intermediate
hmzainjamilhmzainjamil
1

report-generator

Generate formatted reports with charts, tables, and metric summaries from structured data.

Community
Intermediate
weiflycc-cmdweiflycc-cmd

spark-authoring-cli

Manage Microsoft Fabric Spark workloads and author PySpark, Scala, and SQL notebooks via REST API patterns.

Community
Advanced
weiflycc-cmdweiflycc-cmd

spark-consumption-cli

Run interactive PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via the Lakehouse Livy API.

Community
Intermediate