de-production-readiness

What problem does it solve?

This Skill helps data engineering teams turn fragile data pipelines into production-ready systems by enforcing idempotency, reliable retries, SLA monitoring, data freshness checks, observability, and safe change management.

Core Features & Use Cases

• Idempotency Patterns: Use safe upserts, conflict-free inserts, partition overwrite, deterministic job/run IDs, and streaming checkpointing to prevent duplicate loads and make reruns safe.
• Retry Strategies & Error Classification: Implement exponential backoff with jitter, classify transient vs permanent errors, and configure retry behavior for Python, Spark, and Airflow.
• SLA Monitoring, Freshness, and Observability: Add Airflow SLA miss callbacks, Prometheus alerts and pipeline staleness detection, freshness checks (e.g., Soda), structured JSON logging, and OpenTelemetry tracing.
• Data Reconciliation & Graceful Degradation: Validate row counts, checksums, and duplicates; apply circuit breakers, stale-data fallbacks, and skip failed partitions without taking down entire runs.
• Change Management: Perform blue/green table swaps, expand-contract schema migrations, and define rollback procedures to reduce deployment risk.

Quick Start

Use the de-production-readiness skill when you need to make an Airflow or Spark pipeline idempotent and add SLA, freshness, reconciliation, and rollback guardrails before go-live.