fabric-pyspark-perf-remediate

To diagnose slow PySpark performance in Fabric, run prebuilt diagnostic scripts like spark_health_check.py to identify bottlenecks such as stage skew, shuffle issues, or capacity throttling. It provides a structured triage workflow for notebooks and pipelines.

PySpark out of memory errors in Fabric can be diagnosed using the identify_skew.py script to detect stage skew and OOM conditions. The skill provides remediation steps and best practices to resolve memory exhaustion and capacity throttling.

Yes, you can troubleshoot Delta Lake write performance using the delta_table_health.py diagnostic script. It identifies bottlenecks in Delta writes and applies structured remediation workflows to optimize write operations across Fabric data pipelines.

Yes, pyspark is a required dependency. The diagnostic scripts and remediation workflows are built for PySpark workloads running in Microsoft Fabric environments, including Spark Job Definitions and data pipelines.

The best way to identify data skew in Fabric Spark Job Definitions is running the identify_skew.py diagnostic script. It detects stage skew and shuffle bottlenecks, then applies targeted remediation quick fixes to improve runtime reliability.