skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

spark_sql

Optimize and debug production Spark SQL for Hive, lakehouse, and HDFS tables.

14|1|Updated May 5, 2026
Search Tags:#query optimization#window functions#partition pruning#spark sql#hive metastore#etl sql#explain analysis
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill spark-sql
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark_sql
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/spark_sql
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill spark-sql

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you write and optimize production-grade Spark SQL so queries are correct, partition-pruned, and performant on Hive/lakehouse/HDFS tables.

Core Features & Use Cases

  • Query design for correctness: CTE structure, explicit column lists, deterministic windows, and safe join/aggregation patterns.
  • Performance diagnostics & tuning: EXPLAIN FORMATTED/COST guidance, shuffle/joins detection, AQE-aware recommendations, and skew/broadcast handling.
  • Production write safety: Partition-aware filtering, overwrite scope considerations, metastore/HDFS inspection, and DML guardrails.
  • Use case: Review and improve a multi-CTE Spark SQL job that must reliably overwrite the right Hive partitions without exploding join cardinality or scanning unnecessary files.

Quick Start

Ask the AI to review your Spark SQL query for partition pruning and join/window correctness, then suggest the safest performance improvements using EXPLAIN.

Frequently Asked Questions about spark_sql

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize Spark SQL queries for partition pruning on Hive tables?▼

To optimize Spark SQL for partition pruning, use explicit partition-aware filtering in your queries. This ensures filters are pushed down to the storage layer, preventing unnecessary HDFS file scans on Hive tables.

What is the best way to debug Spark SQL performance using EXPLAIN?▼

Debug Spark SQL performance by analyzing EXPLAIN FORMATTED and COST outputs. This Skill diagnoses shuffle and join operations, skew, and broadcast considerations to recommend the safest query improvements.

How do I safely overwrite Hive partitions in Spark SQL without data loss?▼

Safely overwriting Hive partitions requires strict DML planning and overwrite scope considerations. This Skill provides production write safety guardrails to ensure your Spark SQL targets the correct partitions during metastore operations.

How do I fix window function correctness issues in Spark SQL ETL jobs?▼

Fixing window function correctness requires deterministic window definitions and explicit column lists. This Skill reviews your CTE-heavy transformations to ensure safe join and aggregation patterns for large datasets.

Does Adaptive Query Execution (AQE) impact Spark SQL join optimization?▼

Adaptive Query Execution (AQE) significantly impacts Spark SQL join optimization by dynamically managing skew and broadcast joins. This Skill provides AQE-aware recommendations to improve performance and control join cardinality.

Related Skills

Explore More Data & Analytics Skills→
ivanshamaevivanshamaev
14

starrocks-files-ingestion

Ingest and query S3/HDFS files with StarRocks FILES() and Iceberg catalogs.

Community
Intermediate
JeffallanJeffallan
10.9k

spark-engineer

Diagnose and resolve performance bottlenecks in Spark data pipelines.

Community
Advanced
clawdataclawdata
24

spark

Automate building and running Apache Spark batch and streaming data processing jobs.

Official
Intermediate
Awish021Awish021
10

pyspark-databricks

Build and optimize PySpark ETL pipelines on Databricks with Delta Lake.

Community
Intermediate