dataops-airflow-cost-optimizer

What problem does it solve?

This Skill reduces runaway Airflow infrastructure costs caused by inefficient Kubernetes pod scheduling, over-provisioned worker resources, excessive pod churn, and missing retention/cleanup policies that accumulate metadata and logs.

Core Features & Use Cases

• KubernetesPodOperator right-sizing: Tune CPU/memory requests and limits using real observed usage (e.g., P95) so workers aren’t paying for unused capacity.
• Spot and batch optimization: Run batch-heavy workloads on preemptible/spot nodes with tolerations and termination handling to lower compute cost.
• Scheduler and runtime cost control: Consolidate tasks to reduce pod-per-task overhead, right-size the metadata database cleanup cadence, detect over-scheduled DAGs, and implement log lifecycle/retention so storage costs don’t grow unbounded.

Quick Start

Ask an AI to generate an Airflow cost optimization plan by telling it to right-size KubernetesPodOperator resources using P95 metrics, enable S3 log retention, and identify over-scheduled DAGs for schedule reduction.