spark-optimization

Start by enabling Adaptive Query Execution (AQE), right-sizing partitions to 128-256MB each, and using Kryo serialization. Then check the Spark UI for data skew, shuffle spills, and GC pressure to target the specific bottleneck.

Enable spark.sql.adaptive.skewJoin.enabled so AQE splits skewed partitions automatically. For severe skew, apply manual salting by adding a random salt key to the skewed side and exploding the other side across all salt values.

Use a broadcast join when one table is small (under the autoBroadcastJoinThreshold, typically 10-50MB) to avoid shuffling the large side. Sort-merge join is the default for two large tables and handles any data size but requires a shuffle.

Repartition performs a full shuffle to create evenly distributed partitions and can increase or decrease partition count. Coalesce only reduces partitions without a shuffle, making it cheaper but potentially creating uneven partitions.

A common starting point is 8GB executor memory with 2GB overhead, where spark.memory.fraction (0.6) reserves 60% for execution and storage. Monitor GC time and spills in the Spark UI and adjust based on observed pressure.

OOM during shuffle usually comes from too few shuffle partitions, oversized partitions, or skewed keys concentrating data on few executors. Increase spark.sql.shuffle.partitions, enable AQE coalescing, and check partition skew with spark_partition_id counts.