digdag

What problem does it solve?

This Skill provides expert assistance for designing, implementing, and debugging digdag workflows for Treasure Data, simplifying the orchestration of complex data pipelines. It helps users define schedules, manage dependencies, handle errors, and leverage TD-specific operators for efficient and reliable data processing.

Core Features & Use Cases

• Workflow Definition: Guides on structuring .dig files, defining tasks, and configuring schedules (daily, hourly, cron), automating routine data jobs.
• TD Operator Usage: Explains how to use the td> operator for running SQL queries, including inline SQL and external files, with dynamic session variables.
• Error Handling & Retries: Provides patterns for implementing robust error handling with _error: blocks and retry logic with _retry: for resilient workflows that recover from transient issues.
• Use Case: A data analyst needs to automate a daily report that extracts data from multiple TD tables, transforms it, and loads it into a dashboard. This skill helps them define a digdag workflow with scheduled execution, error alerts, and efficient data processing using the td> operator.

Quick Start

Create a simple daily workflow file (e.g., my_workflow.dig)

timezone: Asia/Tokyo schedule: daily>: 03:00:00 +start: echo>: "Daily job for ${session_date} started!"

Run locally in TD environment

td wf run my_workflow.dig