spark-engineer

Develop, debug, and optimize Apache Spark jobs for distributed data processing.

Updated Mar 7, 2026

Search Tags:#data engineering #performance tuning #big data #distributed computing #spark #pyspark

Ready to run

npx skills add https://github.com/shalevamin/The-_Ultimate_agents --skill spark-engineer-shalevamin

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: spark-engineer
Source: https://github.com/shalevamin/The-_Ultimate_agents/tree/main/.claude/skills/spark-engineer
Command: npx skills add https://github.com/shalevamin/The-_Ultimate_agents --skill spark-engineer-shalevamin

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the complexities of building, debugging, and optimizing Apache Spark applications for large-scale data processing, ensuring efficient and performant big data workloads.

Core Features & Use Cases

Spark Job Development: Write and refine PySpark or Scala code for ETL, data transformations, and streaming analytics.
Performance Tuning: Diagnose and resolve performance bottlenecks, optimize cluster configurations, and tune shuffle/caching strategies.
Use Case: You have a PySpark job that is running too slowly. Use this Skill to analyze the Spark UI, identify the bottleneck (e.g., data skew, inefficient joins), and refactor the code for better performance.

Quick Start

Use the spark-engineer skill to write a PySpark job that reads data from 's3://my-bucket/input/' and writes aggregated results to 's3://my-bucket/output/'.

Frequently Asked Questions about spark-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I optimize a slow PySpark job running large-scale data transformations?▼

To optimize a slow PySpark job, analyze the Spark UI to identify bottlenecks like data skew or inefficient joins, then refactor your code by adjusting cluster configurations and tuning shuffle strategies for better performance.

How does Spark handle shuffle operations during distributed data processing?▼

Spark handles shuffle operations by redistributing data across executors during wide transformations like joins and aggregations. Optimizing these shuffle operations is critical for executor memory management and overall pipeline performance.

What is the best way to process .parquet files in a Spark DataFrame pipeline?▼

The best way to process .parquet files in a Spark DataFrame pipeline is to use built-in readers for structured data extraction, apply RDD transformations if needed, and optimize data partitioning to ensure efficient distributed computing.

Can I use Spark SQL for structured streaming analytics on big data workloads?▼

Yes, you can use Spark SQL for structured streaming analytics on big data workloads. It allows you to run standard SQL queries on continuous data streams, enabling real-time ETL processing alongside your batch transformations.

Why does my Spark job fail with out-of-memory errors during executor memory management?▼

Spark jobs fail with out-of-memory errors when executor memory is insufficient for the data partitioning size. Resolving this requires tuning cluster configurations, adjusting executor memory allocations, and optimizing shuffle operations.

Does this approach support writing both ETL transformations and cluster configuration tuning?▼

Yes, this approach supports writing and refining PySpark code for ETL transformations while simultaneously diagnosing cluster configuration issues to tune performance for distributed data processing workloads.