skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

dataops-airflow-cost-optimizer

Optimize Airflow on Kubernetes compute and storage costs via pod right-sizing and spot scheduling.

14|1|Updated May 5, 2026
Search Tags:#kubernetes#cost optimization#airflow#dataops#spot instances#resource right-sizing#keda autoscaling
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill dataops-airflow-cost-optimizer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dataops-airflow-cost-optimizer
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/infra_dataops_group_skills/dataops_airflow_cost_optimizer
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill dataops-airflow-cost-optimizer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill reduces runaway Airflow infrastructure costs caused by inefficient Kubernetes pod scheduling, over-provisioned worker resources, excessive pod churn, and missing retention/cleanup policies that accumulate metadata and logs.

Core Features & Use Cases

  • KubernetesPodOperator right-sizing: Tune CPU/memory requests and limits using real observed usage (e.g., P95) so workers aren’t paying for unused capacity.
  • Spot and batch optimization: Run batch-heavy workloads on preemptible/spot nodes with tolerations and termination handling to lower compute cost.
  • Scheduler and runtime cost control: Consolidate tasks to reduce pod-per-task overhead, right-size the metadata database cleanup cadence, detect over-scheduled DAGs, and implement log lifecycle/retention so storage costs don’t grow unbounded.

Quick Start

Ask an AI to generate an Airflow cost optimization plan by telling it to right-size KubernetesPodOperator resources using P95 metrics, enable S3 log retention, and identify over-scheduled DAGs for schedule reduction.

Frequently Asked Questions about dataops-airflow-cost-optimizer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce Airflow costs on Kubernetes?▼

Reduce Airflow costs on Kubernetes by right-sizing KubernetesPodOperator CPU and memory requests using P95 metrics, scheduling batch tasks on spot nodes, and consolidating tasks to minimize pod-per-task overhead.

How do I right-size Airflow worker pods using observed metrics?▼

Right-size Airflow worker pods by tuning KubernetesPodOperator CPU and memory requests and limits based on real observed P95 usage metrics, ensuring workers do not pay for unused capacity.

Can I run Airflow batch tasks on Kubernetes spot instances?▼

Run Airflow batch tasks on spot instances by applying node tolerations and termination handling to preemptible nodes, lowering compute costs for batch-heavy workloads.

How do I stop unbounded Airflow log and metadata storage growth?▼

Stop unbounded Airflow storage growth by implementing S3 lifecycle log retention policies, right-sizing the metadata database cleanup cadence, and auditing DAG schedules to reduce over-scheduling.

How do I autoscale Airflow workers based on queue depth?▼

Autoscale Airflow workers based on queue depth by implementing KEDA autoscaling, which scales worker resources dynamically according to the task queue size.

Does this Airflow cost optimization approach work with CeleryExecutor?▼

This Airflow cost optimization approach applies to both KubernetesExecutor and CeleryExecutor deployments, targeting pod-per-task overhead and oversized resource requests across these environments.

Related Skills

Explore More Software Engineering Skills→
timurgaleevtimurgaleev
5

benchmark

Detect performance regressions by comparing page load timings and resource metrics.

Community
Advanced
TibsfoxTibsfox
69

kubernetes-patterns

Design and enforce Kubernetes deployment patterns for clusters, Helm charts, service meshes, and autoscaling.

Community
Advanced
dstackaidstackai
2.2k

dstack

Provision and orchestrate GPU workloads across clouds using YAML configurations.

Official
Advanced
snoodleboot-iosnoodleboot-io
2

kubernetes-resource-management

Optimize Kubernetes resource requests, limits, quotas, and priority classes.

Official
Intermediate