skills.rest
lauhon
lauhonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

spark-pod-autosizing

Retrieve Datadog Spark Pod Autosizing recommendations for driver and executor CPU, memory, and storage.

5|2|Updated Mar 9, 2026
Search Tags:#kubernetes#resource optimization#cost optimization#capacity planning#spark#datadog#autosizing
One-click install
npx skills add https://github.com/lauhon/pi --skill spark-pod-autosizing-lauhon
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-pod-autosizing
Source: https://github.com/lauhon/pi/tree/main/skills/spark-pod-autosizing
Command: npx skills add https://github.com/lauhon/pi --skill spark-pod-autosizing-lauhon

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Manually tuning Apache Spark resource configurations through trial and error leads to over-provisioned clusters, wasted cloud spend, and out-of-memory failures. This Skill retrieves data-driven resource recommendations from Datadog's Spark Pod Autosizing API based on actual historical usage metrics. ## Core Features & Use Cases - Resource Recommendations: Query CPU (P75, P95, max percentiles), memory, heap, overhead, and ephemeral storage recommendations for Spark drivers and executors via the pup CLI. - Cost vs Performance Trade-offs: Compare percentile profiles to choose between cost-saving (P75), balanced (P95), or conservative (max) configurations. - Use Case: A data engineer notices an ETL job hitting OOM errors. They query recommendations for the service and shard, discover the driver needs 7168 MiB instead of 4096 MiB, and apply the new values to their SparkApplication manifest. ## Quick Start Ask the agent to get Spark Pod Autosizing recommendations for your service and shard, for example: get resource recommendations for service 'etl-pipeline' with shard 'production'.

Frequently Asked Questions about spark-pod-autosizing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I get Spark resource recommendations from Datadog?▼

Run 'pup spa recommendations --service=<name> --shard=<shard>' with DD_API_KEY, DD_APP_KEY, and DD_SITE environment variables set. The API returns CPU, memory, heap, overhead, and ephemeral storage estimates for both driver and executor.

Should I use P75, P95, or max CPU recommendations for Spark jobs?▼

P95 is recommended for most jobs, balancing cost and performance by covering 95% of workloads. Use P75 when cost is critical and occasional slowdowns are acceptable, and max when job failures are very costly and guaranteed capacity is required.

Why does the Spark Pod Autosizing API return no recommendations?▼

A 404 error means the job has not run enough times to build usage patterns, or the service and shard names do not match your Spark application tags. Verify Datadog is collecting Spark metrics from your jobs.

Can Spark Pod Autosizing automatically apply recommendations to my job?▼

No, the API is read-only and does not modify Spark configurations. You must manually apply the recommended values to your SparkApplication CRD, spark-submit flags, or Spark configuration properties.

What setup is required for Datadog Spark Pod Autosizing?▼

You need the Datadog Agent with the Spark integration enabled, Spark metrics flowing to Datadog, historical data from several job runs, and proper service and shard tags on your Spark applications.

Related Skills

Explore More Software Engineering Skills→
alexmaruccialexmarucci

spark-pod-autosizing

Automate Apache Spark resource allocation on Kubernetes using historical metrics.

Community
Intermediate
yangyongyongyongyangyongyongyong
1

spark-k8s

Diagnose Spark jobs and tasks on Kubernetes via REST APIs.

Community
Advanced
100Thieves-team100Thieves-team

spark-engineer

Writes and optimizes Apache Spark jobs, DataFrame pipelines, and distributed data processing configurations.

Official
Advanced
googlegoogle
19.1k

gke-workload-scaling

Configures HPA and VPA autoscaling for GKE workloads using kubectl and YAML manifests.

Official
Intermediate