spark-engineer

Optimize Spark SQL by tuning spark.sql.shuffle.partitions for your data volume, enabling adaptive query execution, and using broadcast joins for dimension tables under 200MB. Check the Spark UI for shuffle spill and skew, then apply salting or repartitioning where needed.

Handle data skew by salting the skewed key: add a random salt column to both sides of the join, join on the salted key, then drop the salt columns. This distributes hot keys across multiple partitions instead of overloading a single executor.

Use the DataFrame API for structured data processing because it benefits from the Catalyst optimizer and Tungsten execution engine. Reserve RDDs for cases requiring custom partitioners or low-level transformations that DataFrames cannot express.

Out-of-memory failures typically come from calling collect() on large datasets, insufficient executor memory, or severe data skew concentrating records on one partition. Check the Spark UI for spill and skew metrics, then increase executor memory or repartition the data.

Cache a DataFrame only when it is reused multiple times in downstream actions, and materialize it immediately with count() to verify it fits in memory. Call unpersist() when finished, and avoid caching every DataFrame since unnecessary caching wastes memory and can cause spills.