Skills
.Work. You
Rest

spark_sql

Community

Produce fast, safe Spark SQL for production.

Data & Analytics#query optimization#window functions#partition pruning#spark sql#hive metastore#etl sql#explain analysis
Authorivanshamaev
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill helps you write and optimize production-grade Spark SQL so queries are correct, partition-pruned, and performant on Hive/lakehouse/HDFS tables.

Core Features & Use Cases

  • Query design for correctness: CTE structure, explicit column lists, deterministic windows, and safe join/aggregation patterns.
  • Performance diagnostics & tuning: EXPLAIN FORMATTED/COST guidance, shuffle/joins detection, AQE-aware recommendations, and skew/broadcast handling.
  • Production write safety: Partition-aware filtering, overwrite scope considerations, metastore/HDFS inspection, and DML guardrails.
  • Use case: Review and improve a multi-CTE Spark SQL job that must reliably overwrite the right Hive partitions without exploding join cardinality or scanning unnecessary files.

Quick Start

Ask the AI to review your Spark SQL query for partition pruning and join/window correctness, then suggest the safest performance improvements using EXPLAIN.

Dependency Matrix

Required Modules

None required

Components

Standard package

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: spark_sql
Download link: https://github.com/ivanshamaev/de-agent-skills/archive/main.zip#spark-sql

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.