spark-engineer

Optimize Apache Spark applications for performance and pipeline architecture.

9|2|Updated Jan 29, 2026

Search Tags:#data engineering #performance tuning #big data #distributed computing #pyspark #apache spark

Ready to run

npx skills add https://github.com/stephanj/claude-code-collections --skill spark-engineer-stephanj

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: spark-engineer
Source: https://github.com/stephanj/claude-code-collections/tree/main/skills/spark-engineer
Command: npx skills add https://github.com/stephanj/claude-code-collections --skill spark-engineer-stephanj

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the complexity of building and tuning high-performance Apache Spark applications, helping you avoid common pitfalls like memory overflows, data skew, and inefficient shuffle operations.

Core Features & Use Cases

Performance Tuning: Expert guidance on cluster configuration, memory management, and shuffle optimization to ensure your jobs run efficiently.
Pipeline Design: Best practices for choosing between DataFrame and RDD APIs, implementing partitioning strategies, and designing scalable ETL workflows.
Use Case: When your production Spark job is failing due to OutOfMemory errors or running significantly slower than expected, use this Skill to analyze your Spark UI metrics and apply targeted optimizations like salting skewed keys or adjusting shuffle partitions.

Quick Start

Use the spark-engineer skill to analyze my current Spark job performance and suggest optimizations for the observed data skew.

Frequently Asked Questions about spark-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I fix OutOfMemory errors in Apache Spark?▼

To fix OutOfMemory errors in Apache Spark, you must adjust cluster memory management configuration and apply shuffle optimization. Analyzing Spark UI metrics helps identify memory overflows and data skew to apply targeted optimizations like salting skewed keys.

What is the best way to optimize PySpark performance for large-scale ETL?▼

The best way to optimize PySpark performance is by applying best practices for partitioning strategies and pipeline architecture. This ensures scalable ETL workflows and efficient distributed data processing across your cluster resources.

How do I resolve data skew and shuffle bottlenecks in Spark SQL?▼

Resolving data skew and shuffle bottlenecks in Spark SQL requires targeted optimizations like salting skewed keys and adjusting shuffle partitions. This prevents inefficient shuffle operations and ensures balanced distributed data processing.

When should I use DataFrame API versus RDD operations in big data pipelines?▼

Choosing between DataFrame API and RDD operations depends on your pipeline design requirements for distributed computing. DataFrames generally offer better optimization for large-scale ETL development, while RDDs provide lower-level control for specific big data processing tasks.

Can I use this approach for cluster resource management and performance tuning?▼

Yes, this approach provides expert guidance specifically for cluster resource management and performance tuning. It helps configure cluster resources effectively to avoid common pitfalls like memory overflows and data skew during distributed data processing.