skills.rest
yangyongyongyong
yangyongyongyongCOMMUNITY
Explore all skills by this author →
GitHub

spark-k8s

Diagnose Spark jobs and tasks on Kubernetes via REST APIs.

1|Updated Apr 11, 2026
Search Tags:#Kubernetes#REST API#Spark#performance analysis#job diagnostics
One-click install
npx skills add https://github.com/yangyongyongyong/skills --skill spark-k8s
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-k8s
Source: https://github.com/yangyongyongyong/skills/tree/main/spark-k8s
Command: npx skills add https://github.com/yangyongyongyong/skills --skill spark-k8s

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires httpx, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a comprehensive toolset for diagnosing Spark jobs and tasks on Kubernetes, allowing you to analyze job status, stages, executors, and more using REST APIs.

Core Features & Use Cases

  • Job Status and History: View job status, duration, and stage information.
  • Executor and Task Analysis: Inspect executor health, task performance, and skew.
  • Resource Utilization: Assess resource usage, such as input, output, and shuffle data.
  • SQL Execution: Analyze SQL execution metrics, including failures and plans.
  • Use Case: If you are facing performance issues with a Spark job running on Kubernetes, this Skill can help you identify bottlenecks such as slow stages, resource allocation, or executor failures.

Quick Start

Use the spark-k8s skill to list the applications in the Spark History Server.

Frequently Asked Questions about spark-k8s

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose Spark job performance issues on Kubernetes?▼

To diagnose Spark job performance issues on Kubernetes, you can query the Spark UI or History Server via REST APIs to analyze job status, executor health, and task skew. This identifies bottlenecks like slow stages or resource allocation problems.

How do I monitor Spark executor health and task skew in a Kubernetes environment?▼

You can monitor Spark executor health and task skew on Kubernetes by interacting with REST APIs to inspect executor metrics and task performance. This allows you to assess resource utilization including input, output, and shuffle data.

Can I analyze Spark SQL execution plans and failures using REST APIs?▼

Yes, you can analyze Spark SQL execution metrics including failures and execution plans by sending REST API requests to the Spark History Server. This helps pinpoint SQL query bottlenecks within your Spark jobs.

Do I need access to the Spark History Server to analyze jobs on Kubernetes?▼

Yes, you need access to either the Spark UI or the Spark History Server to retrieve application data. You also need Python scripts utilizing the httpx dependency to interact with the REST APIs for job analysis.

What's the best way to list Spark applications running on Kubernetes?▼

The best way to list Spark applications on Kubernetes is to query the Spark History Server through REST APIs. This approach retrieves job status, duration, and stage information efficiently for active and completed applications.

Related Skills

Explore More Software Engineering Skills→
delta-and-betadelta-and-beta

engineer-mcp-server

Install and configure the Braze MCP server for Braze REST API access.

Community
Advanced
PatrickGallucciPatrickGallucci
16

fabric-pyspark-perf-remediate

Diagnose and remediate Spark performance issues in Microsoft Fabric PySpark workloads.

Community
Advanced
cnoe-iocnoe-io
392

cluster-resource-health

Query EKS clusters via kubectl and report pod, node, and resource health.

Official
Intermediate
nksethnkseth

spark-engineer

Optimize Spark DataFrame and RDD pipelines with performance tuning.

Community
Advanced