Optimize Apache Spark and PySpark jobs with performance tuning guidance.
npx skills add https://github.com/brunoldqueiroz/marvin --skill spark-expert
Please help me install this Agent Skill. Skill: spark-expert Source: https://github.com/brunoldqueiroz/marvin/tree/main/.claude/skills/spark-expert Command: npx skills add https://github.com/brunoldqueiroz/marvin --skill spark-expert
This Skill helps users optimize Apache Spark and PySpark jobs, resolve performance bottlenecks, and implement best practices for big data processing.
OPTIMIZE and MERGE operations, including ZORDER and Liquid Clustering.withColumn loops and Python UDFs, suggesting efficient alternatives.Use the spark-expert skill to optimize a slow PySpark job that is experiencing shuffle spills.
High-intent search queries and answers about installing and using this skill.
To optimize PySpark jobs and reduce shuffle spills, you must tune partitioning, manage memory effectively, and apply efficient coding patterns to improve overall execution speed and resource utilization.
The best way to optimize Delta Lake MERGE operations is by applying best practices for OPTIMIZE commands, utilizing ZORDER, and implementing Liquid Clustering to accelerate data skipping and query performance.
Spark executor OOM errors occur due to inadequate memory management and inefficient partitioning, requiring you to diagnose common anti-patterns and adjust resource allocation to resolve the bottlenecks.
Yes, tuning Apache Spark performance requires an understanding of distributed computing concepts and the Spark execution engine to effectively address shuffles, partitioning, and memory management.
Common PySpark anti-patterns that slow down big data processing include using withColumn loops and Python UDFs, which can be replaced with efficient native alternatives to prevent slow plan generation.
hiddink-aiProvide architectural guidance and performance tuning for Apache Spark 4.0.2 jobs.
Gami19Design, optimize, and operate Delta Lake tables across Bronze, Silver, and Gold layers.
PriyanshKuniyalOptimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning.
ianpojmanDiagnose Spark/EMR job failures and optimize performance bottlenecks.