skills.rest
PatrickGallucci
PatrickGallucciCOMMUNITY
Explore all skills by this author →
GitHub

fabric-pandas-perf-remediate

Diagnose pandas performance bottlenecks in Fabric Spark notebooks.

16|1|Updated Feb 10, 2026
Search Tags:#performance#memory#diagnostics#pandas#spark#fabric
One-click install
npx skills add https://github.com/PatrickGallucci/fabric-skills --skill fabric-pandas-perf-remediate
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: fabric-pandas-perf-remediate
Source: https://github.com/PatrickGallucci/fabric-skills/tree/main/skills/fabric-pandas-perf-remediate
Command: npx skills add https://github.com/PatrickGallucci/fabric-skills --skill fabric-pandas-perf-remediate

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires Az.Accounts, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill helps diagnose and optimize pandas performance in Microsoft Fabric Spark notebooks, reducing driver memory pressure, preventing toPandas() bottlenecks, and enabling efficient data workflows.

Core Features & Use Cases

  • Diagnose pandas API on Spark bottlenecks and memory issues in Fabric notebooks.
  • Recommend Spark configuration tuning, data sizing strategies, and memory profiling workflows.
  • Typical use cases include large pandas-on-Spark transformations, driver memory optimization, and end-to-end performance troubleshooting.

Quick Start

Install the diagnostic tooling, run the diagnostic script, and review the generated report for actionable recommendations.

Frequently Asked Questions about fabric-pandas-perf-remediate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose pandas performance bottlenecks in Fabric Spark notebooks?▼

To diagnose pandas performance bottlenecks in Fabric Spark notebooks, run the diagnostic script to analyze memory profiling, UDF usage, and Delta reads, generating a report with actionable recommendations.

Why does toPandas() cause driver memory pressure in Microsoft Fabric?▼

Calling toPandas() causes driver memory pressure by transferring large Spark datasets directly to the driver node. This skill identifies these bottlenecks and recommends data sizing strategies to prevent inefficient data transfers.

What is the best way to tune Spark configuration for large pandas-on-Spark workloads?▼

The best way to tune Spark configuration for pandas-on-Spark workloads is applying adaptive query execution, autotune, Arrow transfer optimization, and resource profiles guided by diagnostic memory profiling reports.

Can I use this diagnostic tool to optimize UDF usage and Delta reads in Fabric?▼

Yes, you can use this tool to optimize UDF usage and Delta reads in Fabric. It diagnoses inefficient data transfers and recommends Spark configuration tuning to improve end-to-end performance.

Do I need Az.Accounts to run pandas memory profiling diagnostics in Fabric?▼

Yes, you need the Az.Accounts dependency installed to authenticate and run pandas memory profiling diagnostics across your Fabric Spark notebooks environment.

Related Skills

Explore More Data & Analytics Skills→
PatrickGallucciPatrickGallucci
16

fabric-spark-perf-remediate

Diagnose and remediate Fabric Spark performance problems across workloads.

Community
Advanced
haxlyshaxlys

spark-optimization

Optimize Spark jobs with partitioning, memory tuning, and shuffle optimization.

Community
Advanced
PatrickGallucciPatrickGallucci
16

fabric-delta-spark-perf

Diagnose Delta Lake and Spark performance bottlenecks in Fabric Lakehouses.

Community
Advanced
K-Dense-AIK-Dense-AI
33.0k

polars

Replace pandas with Polars for in-memory DataFrame processing in Python and Rust.

Official
Advanced