spark-engineer

Optimize Spark pipelines by using DataFrame API over RDDs, defining explicit schemas, partitioning data to 200-1000 partitions per executor core, caching strategically, employing broadcast joins for small tables under 200MB, and monitoring Spark UI metrics for shuffle, spill, and garbage collection to identify bottlenecks.

Partition Spark data by targeting 200-1000 partitions per executor core to balance parallelism and overhead. Use custom partitioning or salting techniques to mitigate data skew, and leverage the DataFrame API for efficient repartitioning aligned with your ETL workflow requirements.

Use Spark SQL and DataFrame API for most transformations because they enable Catalyst optimizer to improve performance automatically. Reserve RDDs only for unstructured data or low-level transformations where DataFrames cannot express your logic efficiently.

Address data skew by salting keys to distribute uneven partitions, applying custom partitioning strategies, or using broadcast joins for dimension tables. Monitor Spark UI for unbalanced shuffle metrics and task durations to detect and resolve skew in production pipelines.

Yes, Spark SQL works for both batch ETL and structured streaming pipelines. Use consistent schema definitions and DataFrame transformations across both patterns to leverage the same optimization engine and simplify production-grade pipeline maintenance.

Cache DataFrames judiciously by caching only reused intermediate results, monitor Spark UI for spill metrics, and size your executor memory and partitions to keep working sets in memory. Avoid caching very large datasets; instead, repartition and persist only critical transformation points.