skills.rest
weiflycc-cmd
weiflycc-cmdCOMMUNITY
Explore all skills by this author →
GitHub

spark-consumption-cli

Run interactive PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via the Lakehouse Livy API.

Updated Jun 1, 2026
Search Tags:#pyspark#spark sql#microsoft fabric#delta time travel#lakehouse analytics#livy sessions#cross-lakehouse joins
One-click install
npx skills add https://github.com/weiflycc-cmd/skills-for-fabric --skill spark-consumption-cli-weiflycc-cmd
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-consumption-cli
Source: https://github.com/weiflycc-cmd/skills-for-fabric/tree/main/plugins/fabric-skills/skills/spark-consumption-cli
Command: npx skills add https://github.com/weiflycc-cmd/skills-for-fabric --skill spark-consumption-cli-weiflycc-cmd

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the gap for data engineers and analysts who need to run advanced Python-based Spark analytics on Microsoft Fabric lakehouse data that is not possible with simple SQL queries, including processing unstructured/JSON data, performing Delta time-travel, running cross-lakehouse joins, and executing complex DataFrame operations, by providing Fabric-specific Livy API session management and CLI integration patterns.

Core Features & Use Cases

  • Automated Livy Session Management: Handles creation, reuse, and cleanup of Fabric Lakehouse Livy API sessions for interactive PySpark/Spark SQL workloads, eliminating manual configuration and reducing unnecessary resource consumption.
  • Fabric-Optimized Spark Patterns: Provides pre-built, Fabric-specific patterns for cross-workspace lakehouse queries, Delta time-travel, unstructured JSON processing, and DataFrame operations aligned with OneLake architecture requirements.
  • Real-World Use Case: A data engineer needs to analyze JSON clickstream data in a Fabric lakehouse, join it with Delta table data from a separate workspace, and compare historical data versions via time-travel queries — all using interactive PySpark without setting up or managing notebook infrastructure.

Quick Start

Use the spark-consumption-cli skill to perform interactive PySpark analysis of your Fabric lakehouse data, including cross-lakehouse joins, Delta time-travel queries, and unstructured JSON processing, via a managed Livy session.

Frequently Asked Questions about spark-consumption-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run interactive PySpark analytics on Microsoft Fabric lakehouse data without using notebooks?▼

You can run interactive PySpark on Fabric lakehouse data without notebooks by using the Lakehouse Livy API to create interactive sessions. This approach executes ad-hoc Spark workloads directly, eliminating the need for manual cluster configuration or notebook infrastructure.

Can I perform cross-workspace lakehouse joins in Fabric using PySpark?▼

Yes, you can perform cross-workspace lakehouse joins in Fabric using PySpark via the Livy API. This allows you to query and merge Delta table data from separate workspaces using Fabric-optimized DataFrame operations aligned with the OneLake architecture.

What is Delta time-travel and how does it work for Fabric lakehouse queries?▼

Delta time-travel in Fabric lakehouse queries allows you to access and compare historical data versions of Delta tables. By leveraging PySpark via the Livy API, you can execute queries against previous states of your data without needing manual snapshot management.

Does the Fabric Livy API support processing unstructured JSON data with PySpark?▼

Yes, the Fabric Livy API supports processing unstructured JSON data using PySpark DataFrame operations. This enables you to analyze raw JSON clickstream data stored in a lakehouse, applying Python-based data quality checks without relying on simple SQL queries.

Do I need to manually configure Spark clusters to run ad-hoc Spark SQL workloads in Microsoft Fabric?▼

No, you do not need to manually configure Spark clusters to run ad-hoc workloads in Microsoft Fabric. The Livy API handles automated session management, including creation, reuse, and cleanup, so you can execute Spark SQL directly.

Related Skills

Explore More Data & Analytics Skills→
patrikboroschpatrikborosch
14

spark-consumption-cli

Run PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via Livy API sessions.

Community
Intermediate
microsoftmicrosoft
952

spark-consumption-cli

Run interactive PySpark and Spark SQL queries on Fabric Lakehouse Livy sessions.

Official
Advanced
rritecrritec
3

spark-consumption-cli

Analyze Microsoft Fabric lakehouse data with interactive PySpark sessions and Spark SQL.

Community
Advanced
weiflycc-cmdweiflycc-cmd

spark-authoring-cli

Manage Microsoft Fabric Spark workloads and author PySpark, Scala, and SQL notebooks via REST API patterns.

Community
Advanced