skills.rest
patrikborosch
patrikboroschCOMMUNITY
Explore all skills by this author →
GitHub

spark-consumption-cli

Run PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via Livy API sessions.

14|3|Updated Mar 13, 2026
Search Tags:#pyspark#spark sql#microsoft fabric#delta time travel#lakehouse analytics#livy sessions#cross-lakehouse joins
One-click install
npx skills add https://github.com/patrikborosch/AnalyticsPlatformAgents --skill spark-consumption-cli-patrikborosch
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-consumption-cli
Source: https://github.com/patrikborosch/AnalyticsPlatformAgents/tree/main/creator/skills/spark-consumption-cli
Command: npx skills add https://github.com/patrikborosch/AnalyticsPlatformAgents --skill spark-consumption-cli-patrikborosch

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the need for manual management of Fabric Lakehouse Livy API sessions to run advanced PySpark and Spark SQL analytics that exceed standard SQL capabilities, such as Delta time-travel, cross-lakehouse joins, and unstructured data analysis.

Core Features & Use Cases

  • Managed Livy Session Lifecycle: Automatically checks for existing idle sessions, creates new Spark sessions on the Starter Pool when needed, and cleans up idle sessions to avoid resource waste.
  • Advanced Spark Analytics Support: Enables use of PySpark DataFrames, Delta Lake features (time travel, versioning), cross-workspace lakehouse joins, and analysis of unstructured/JSON data.
  • Use Case: A data engineer can use this skill to audit historical changes to a Delta table using time-travel queries, or join customer data from a lakehouse in the sales workspace with transaction data from a lakehouse in the finance workspace for a unified customer analytics report.

Quick Start

Use the spark-consumption-cli skill to run a PySpark analysis of historical Delta table versions across your Fabric lakehouses.

Frequently Asked Questions about spark-consumption-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run PySpark analytics on Microsoft Fabric lakehouse tables?▼

You can run PySpark analytics on Microsoft Fabric lakehouse tables by using managed Livy API sessions to execute PySpark DataFrames and Spark SQL queries. This approach handles authentication, workspace discovery, and Spark session lifecycle automatically.

How do I query historical Delta table versions in a Fabric lakehouse?▼

Querying historical Delta table versions in a Fabric lakehouse requires Delta time-travel capabilities. Using PySpark within managed Livy sessions, you can audit historical changes and access specific table versions that exceed standard SQL query functions.

Can I join data across different Fabric lakehouses and workspaces?▼

Yes, you can perform cross-lakehouse joins across different Fabric workspaces. By utilizing PySpark DataFrames within Livy sessions, you can join data from separate lakehouses, such as sales and finance workspaces, for unified analytics reporting.

Do I need to manually manage Spark sessions when running Spark SQL in Fabric?▼

No, you do not need to manually manage Spark sessions for Spark SQL in Fabric. Managed Livy session lifecycle handling automatically checks for idle sessions, creates new ones on the Starter Pool, and cleans up resources to prevent waste.

How do I analyze unstructured and JSON data in a Fabric lakehouse?▼

Analyzing unstructured and JSON data in a Fabric lakehouse involves using PySpark DataFrames. This method allows you to parse and process complex data formats beyond standard SQL capabilities, executing directly within interactive Livy API sessions.

What is the best way to perform data quality checks on Fabric lakehouse tables?▼

Performing data quality checks on Fabric lakehouse tables is best achieved using PySpark DataFrames. This allows you to validate data integrity and apply complex rules within managed Spark sessions, handling session cleanup automatically.

Related Skills

Explore More Data & Analytics Skills→
rritecrritec
3

spark-consumption-cli

Analyze Microsoft Fabric lakehouse data with interactive PySpark sessions and Spark SQL.

Community
Advanced
microsoftmicrosoft
952

spark-consumption-cli

Run interactive PySpark and Spark SQL queries on Fabric Lakehouse Livy sessions.

Official
Advanced
weiflycc-cmdweiflycc-cmd

spark-authoring-cli

Manage Microsoft Fabric Spark workloads and author PySpark, Scala, and SQL notebooks via REST API patterns.

Community
Advanced
weiflycc-cmdweiflycc-cmd

spark-consumption-cli

Run interactive PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via the Lakehouse Livy API.

Community
Intermediate