workflow-management

What problem does it solve?

This Skill provides expert assistance for managing, debugging, monitoring, and optimizing Treasure Data workflows (Digdag pipelines). It helps users quickly identify and resolve failures, improve performance, and implement best practices for reliable data operations, reducing downtime and operational complexity.

Core Features & Use Cases

• Debugging & Troubleshooting: Guides on investigating failed sessions, checking logs, and identifying root causes for common workflow errors, accelerating problem resolution.
• Performance Optimization: Offers strategies for optimizing slow queries, managing resource limits, and improving overall workflow execution speed, saving compute costs and time.
• Alerting & Monitoring: Provides patterns for setting up notifications (Slack, email) on workflow failures and implementing data quality checks, ensuring proactive issue detection.
• Use Case: A critical daily ETL workflow fails unexpectedly. This skill guides the user through checking session logs, identifying a slow query, and implementing a retry mechanism with alerts, ensuring the data pipeline is quickly restored and future failures are proactively managed.

Quick Start

Check recent workflow failures

digdag sessions <workflow_name> --failed --last 20

View logs for a specific failed session and task

digdag log <session_id> +task_name