Skills
.Work. You
Rest

fabric-spark

Community

Optimize and streamline PySpark operations in Microsoft Fabric notebooks.

Data & Analytics#data processing#lakehouse#PySpark#Microsoft Fabric#notebook optimization
Authorwardawgmalvicious
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill addresses the complexities of using PySpark within Microsoft Fabric notebooks, ensuring efficient and effective data processing by providing best practices, constraints, and configurations.

Core Features & Use Cases

  • Constraint Handling: Ensures adherence to Fabric's no-external-HTTP constraint and ABFS URI formatting.
  • Runtime Configuration: Offers guidance on notebookutils.runtime.context for identity lookups and spark.conf.* for session tuning.
  • Lakehouse Setup: Details schema immutability, cross-lakehouse names, and table maintenance.
  • Notebook REST API: Explains API usage for notebook creation, execution, and metadata updates.
  • Auto-Restart: Enables automatic notebook restarts for pipeline-driven runs.
  • Use Case: A data engineer uses this Skill to configure a PySpark job in a Fabric notebook, ensuring efficient data processing and minimizing errors.

Quick Start

Use the fabric-spark skill to optimize a PySpark job in a Fabric notebook, following the provided guidelines for lakehouse setup and configuration.

Dependency Matrix

Required Modules

None required

Components

scriptsreferences

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: fabric-spark
Download link: https://github.com/wardawgmalvicious/claude-config/archive/main.zip#fabric-spark

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.