spark-engineer

Optimize Spark DataFrame and RDD pipelines with performance tuning.

Updated May 14, 2026

Search Tags:#streaming #etl #spark #pyspark #data-processing #distributed-computing #spark-sql

Ready to run

npx skills add https://github.com/nkseth/copilot-dev-skills --skill spark-engineer-nkseth

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: spark-engineer
Source: https://github.com/nkseth/copilot-dev-skills/tree/main/skills/spark-engineer
Command: npx skills add https://github.com/nkseth/copilot-dev-skills --skill spark-engineer-nkseth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Spark engineering solves the complexity of building, tuning, and maintaining high-performance Spark workloads by providing structured guidance for DataFrame and RDD workflows, memory management, and streaming patterns.

Core Features & Use Cases

Writing DataFrame transformations, optimizing Spark SQL queries, and implementing efficient UDF-friendly pipelines.
Designing scalable ETL and analytics pipelines, tuning shuffle and executor memory, and handling data format like Parquet.
Building structured streaming applications with fault tolerance and throughput optimization.
Use cases include batch ETL at scale, real-time analytics, and large-scale data processing on distributed clusters.

Quick Start

Run a sample Spark job to transform input data and write the results to Parquet.

Frequently Asked Questions about spark-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I optimize Spark SQL queries and tune shuffle memory for large-scale data processing?▼

To optimize Spark SQL and shuffle memory, inspect Spark UI performance metrics, define explicit schemas, and apply partitioning best practices to streamline large-scale data processing workflows.

What is the best way to build fault-tolerant structured streaming pipelines on distributed clusters?▼

Building structured streaming pipelines requires implementing fault tolerance and throughput optimization using PySpark or Scala, ensuring efficient data processing across distributed clusters.

Do I need to know Scala or PySpark to use this for DataFrame and RDD pipeline development?▼

Yes, you need knowledge of PySpark or Scala to build DataFrame and RDD pipelines, along with familiarity with Spark UI inspection for performance metrics and memory tuning.

How do I process Parquet files and design scalable ETL pipelines in Spark?▼

Processing Parquet files and designing scalable ETL pipelines involves writing DataFrame transformations, optimizing UDF-friendly pipelines, and adhering to executor memory and partitioning best practices.

Why does my Spark job fail during large-scale batch ETL and how can I tune executor memory?▼

Spark jobs fail during batch ETL when executor memory and shuffle configurations are suboptimal; tune memory by inspecting Spark UI metrics and applying partitioning best practices for distributed clusters.