spark-expert

Optimize Apache Spark and PySpark jobs with performance tuning guidance.

Updated Aug 27, 2026

Search Tags:#optimization #data engineering #performance tuning #big data #spark #pyspark #delta lake

Ready to run

npx skills add https://github.com/brunoldqueiroz/marvin --skill spark-expert

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: spark-expert
Source: https://github.com/brunoldqueiroz/marvin/tree/main/.claude/skills/spark-expert
Command: npx skills add https://github.com/brunoldqueiroz/marvin --skill spark-expert

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps users optimize Apache Spark and PySpark jobs, resolve performance bottlenecks, and implement best practices for big data processing.

Core Features & Use Cases

Performance Tuning: Provides guidance on optimizing shuffle operations, partitioning, and memory management.
Delta Lake Optimization: Offers best practices for Delta Lake OPTIMIZE and MERGE operations, including ZORDER and Liquid Clustering.
Code Best Practices: Identifies anti-patterns like withColumn loops and Python UDFs, suggesting efficient alternatives.
Troubleshooting: Assists in diagnosing common errors like executor OOMs and slow plan generation.

Quick Start

Use the spark-expert skill to optimize a slow PySpark job that is experiencing shuffle spills.

Frequently Asked Questions about spark-expert

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I optimize PySpark jobs to reduce shuffle spills and improve execution speed?▼

To optimize PySpark jobs and reduce shuffle spills, you must tune partitioning, manage memory effectively, and apply efficient coding patterns to improve overall execution speed and resource utilization.

What is the best way to optimize Delta Lake MERGE operations and handle large datasets?▼

The best way to optimize Delta Lake MERGE operations is by applying best practices for OPTIMIZE commands, utilizing ZORDER, and implementing Liquid Clustering to accelerate data skipping and query performance.

Why does my Spark job fail with executor OOM errors during large data processing?▼

Spark executor OOM errors occur due to inadequate memory management and inefficient partitioning, requiring you to diagnose common anti-patterns and adjust resource allocation to resolve the bottlenecks.

Do I need to understand distributed computing concepts to tune Apache Spark performance?▼

Yes, tuning Apache Spark performance requires an understanding of distributed computing concepts and the Spark execution engine to effectively address shuffles, partitioning, and memory management.

What are common PySpark anti-patterns that slow down big data processing?▼

Common PySpark anti-patterns that slow down big data processing include using withColumn loops and Python UDFs, which can be replaced with efficient native alternatives to prevent slow plan generation.