kubernetes-data-platform
CommunityRun Spark and Airflow reliably on K8s
Authorivanshamaev
Version1.0.0
Installs0
System Documentation
What problem does it solve?
You need a practical, production-ready way to run data processing workloads on Kubernetes while handling Spark-on-K8s specifics and Airflow Helm configuration, including RBAC, scheduling, resources, and troubleshooting.
Core Features & Use Cases
- Spark on Kubernetes setup: configure cluster mode spark-submit, Kubernetes namespaces, driver/executor service account access, pod templates, event logging, and dynamic allocation.
- Airflow on Kubernetes via Helm: deploy Airflow using the Helm chart with KubernetesExecutor, remote logging to object storage, git-sync for DAGs, metadata DB configuration, and resource tuning.
- Operational guardrails: namespace quotas/limit ranges, secrets management, common debugging commands, and anti-patterns that prevent common production failures.
- Use case: migrate an ETL pipeline so Airflow schedules Spark jobs on Kubernetes, stores logs/events remotely, deploys DAGs from Git, and enforces safe resource limits to avoid pod exhaustion.
Quick Start
Ask the Kubernetes data platform skill to generate an end-to-end KubernetesExecutor + Spark-on-K8s setup plan including RBAC, Helm values, git-sync DAG configuration, and pod resource/quotas for your namespace.
Dependency Matrix
Required Modules
None requiredComponents
Standard package💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: kubernetes-data-platform Download link: https://github.com/ivanshamaev/de-agent-skills/archive/main.zip#kubernetes-data-platform Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.