Verified:GitHub

des-orchestration-observability

Design orchestration and observability specifications for data engineering workflows.

2|Updated May 20, 2026

Search Tags:#orchestration #observability #quality gates #data pipelines #SLA monitoring #retries and recovery #run evidence

Ready to run

npx skills add https://github.com/DKSang/DES-SKILL --skill des-orchestration-observability

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: des-orchestration-observability
Source: https://github.com/DKSang/DES-SKILL/tree/main/skills/des-orchestration-observability
Command: npx skills add https://github.com/DKSang/DES-SKILL --skill des-orchestration-observability

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill prevents data-pipeline failures from becoming silent, unrecoverable incidents by specifying how orchestration and observability must work before implementation starts.

Core Features & Use Cases

Workflow orchestration design: define workflow inventory, dependencies, triggers, schedules, and quality gate integration for ingestion through Bronze/Silver/Gold.
Operational safety & recovery planning: specify retries, timeouts, failure handling, backfills/replays, and late/correction behavior so pipelines can be restarted and audited safely.
Observability & audit readiness: inventory monitoring signals (freshness/SLA, volume, quality results, runtime, cost) and define run evidence for debugging, governance, and incident response.

Quick Start

Use the des-orchestration-observability skill to produce the Phase 15 Orchestration and Observability Specification by validating scheduling, dependency ordering, gates, retry/recovery behavior, and the required operational evidence for your data pipelines.

Frequently Asked Questions about des-orchestration-observability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design orchestration and observability for safe data pipelines?▼

Data pipeline orchestration and observability design requires defining workflow dependencies, quality gate placement, retry handling, and SLA monitoring to prevent silent failures and ensure recoverable operations across ingestion, Bronze, Silver, and Gold layers.

What is the best way to handle retries and recovery in scheduled data pipelines?▼

Retries and recovery for data pipelines involve specifying timeout limits, automated failure handling, backfill mechanisms, and replay procedures to ensure pipelines can be safely restarted and audited without losing data integrity or run evidence.

How do I set up quality gates in end-to-end data engineering workflows?▼

Quality gates in data engineering workflows are set by defining validation checkpoints between pipeline stages, integrating them with the orchestration schedule, and monitoring quality results as operational signals before data progresses from Bronze to Silver and Gold layers.

What monitoring signals do I need for data pipeline SLA monitoring and audit readiness?▼

SLA monitoring and audit readiness require inventorying operational signals including data freshness, volume, quality results, runtime, and cost, alongside capturing run evidence for debugging, governance, and incident response during pipeline execution.

Do I need validated upstream planning artifacts to design pipeline observability specifications?▼

Yes, designing pipeline observability specifications requires validated upstream planning artifacts and Phase 14 handoff inputs to accurately define scheduling, dependency ordering, and failure handling for the Phase 15 orchestration specification.

Can I use this orchestration design approach for backfills and late data corrections?▼

Yes, orchestration design for data pipelines explicitly accommodates backfill and replay operations by specifying correction behaviors and late data handling, ensuring historical data reprocessing remains auditable and safely recoverable within the scheduled workflow.