skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

kubernetes-data-platform

Deploy Kubernetes-based data platforms for Spark and Airflow workloads.

14|1|Updated May 5, 2026
Search Tags:#rbac#kubernetes#resource management#helm#airflow#spark#kubernetesexecutor
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill kubernetes-data-platform
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kubernetes-data-platform
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/kubernetes_data
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill kubernetes-data-platform

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

You need a practical, production-ready way to run data processing workloads on Kubernetes while handling Spark-on-K8s specifics and Airflow Helm configuration, including RBAC, scheduling, resources, and troubleshooting.

Core Features & Use Cases

  • Spark on Kubernetes setup: configure cluster mode spark-submit, Kubernetes namespaces, driver/executor service account access, pod templates, event logging, and dynamic allocation.
  • Airflow on Kubernetes via Helm: deploy Airflow using the Helm chart with KubernetesExecutor, remote logging to object storage, git-sync for DAGs, metadata DB configuration, and resource tuning.
  • Operational guardrails: namespace quotas/limit ranges, secrets management, common debugging commands, and anti-patterns that prevent common production failures.
  • Use case: migrate an ETL pipeline so Airflow schedules Spark jobs on Kubernetes, stores logs/events remotely, deploys DAGs from Git, and enforces safe resource limits to avoid pod exhaustion.

Quick Start

Ask the Kubernetes data platform skill to generate an end-to-end KubernetesExecutor + Spark-on-K8s setup plan including RBAC, Helm values, git-sync DAG configuration, and pod resource/quotas for your namespace.

Frequently Asked Questions about kubernetes-data-platform

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy Airflow on Kubernetes with Helm using KubernetesExecutor?▼

To deploy Airflow on Kubernetes with Helm, configure the KubernetesExecutor, remote logging to object storage, git-sync for DAGs, and metadata DB settings. This ensures reliable scheduling and isolates task execution into individual pods.

How do I configure Spark on Kubernetes with dynamic allocation and RBAC?▼

Configure Spark on Kubernetes by setting up driver and executor service accounts with proper RBAC, applying pod templates, and enabling dynamic allocation. This aligns Spark resource requests with cluster autoscaling to prevent pod exhaustion.

Can I use Airflow KubernetesPodOperator to submit Spark jobs on Kubernetes?▼

Yes, you can use the Airflow KubernetesPodOperator to submit Spark jobs on Kubernetes. This setup allows Airflow to schedule Spark workloads in cluster mode while enforcing namespace resource quotas and limit ranges.

Why are my Spark executor pods pending or failing to schedule on Kubernetes?▼

Spark executor pods fail to schedule on Kubernetes due to improper RBAC permissions, exceeded namespace resource quotas, or misconfigured dynamic allocation. Troubleshoot by checking pod events and aligning resource requests with cluster limits.

Do I need namespace quotas and limit ranges to run Spark and Airflow on Kubernetes?▼

Yes, namespace quotas and limit ranges are required operational guardrails for running Spark and Airflow on Kubernetes. They enforce safe resource limits per namespace, preventing pod exhaustion and common production scheduling failures.

What is the best way to sync Airflow DAGs from Git on Kubernetes?▼

The best way to sync Airflow DAGs from Git on Kubernetes is using the git-sync configuration provided by the Airflow Helm chart. This pulls DAG files directly from your repository into the Airflow scheduler and webserver pods.

Related Skills

Explore More Software Engineering Skills→
rohitg00rohitg00
944

k8s-deploy

Deploy Kubernetes workloads with progressive delivery strategies and health checks.

Community
Advanced
timbuchingertimbuchinger
1

azure-troubleshoot

Diagnose Azure infrastructure issues across VMs, AKS, and storage.

Community
Basic
theandyalvarez7-rubytheandyalvarez7-ruby

senior-data-engineer

Designs scalable batch and streaming data pipelines with Airflow, dbt, Spark, and Kafka.

Community
Advanced
2nth-ai2nth-ai
1

tech/google/cloud/compute

Deploy and manage Google Cloud resources using gcloud CLI.

Official
Advanced