Orchestrate Treasure Data Digdag workflows with td> operators and session variables.
npx skills add https://github.com/treasure-data/td-skills --skill digdag
Please help me install this Agent Skill. Skill: digdag Source: https://github.com/treasure-data/td-skills/tree/main/workflow-skills/digdag Command: npx skills add https://github.com/treasure-data/td-skills --skill digdag
This Skill provides expert assistance for designing, implementing, and debugging digdag workflows for Treasure Data, simplifying the orchestration of complex data pipelines. It helps users define schedules, manage dependencies, handle errors, and leverage TD-specific operators for efficient and reliable data processing.
.dig files, defining tasks, and configuring schedules (daily, hourly, cron), automating routine data jobs.td> operator for running SQL queries, including inline SQL and external files, with dynamic session variables._error: blocks and retry logic with _retry: for resilient workflows that recover from transient issues.td> operator.timezone: Asia/Tokyo schedule: daily>: 03:00:00 +start: echo>: "Daily job for ${session_date} started!"
td wf run my_workflow.dig
High-intent search queries and answers about installing and using this skill.
Digdag orchestrates scheduled data pipelines by defining tasks in `.dig` files with cron or daily schedules, executing them automatically at specified times. Use the `schedule:` directive to set frequency (daily, hourly, cron syntax) and structure tasks with `+task` prefixes to run sequentially or in parallel, enabling hands-off automation of ETL jobs.
The `td>` operator executes SQL queries directly within digdag workflows. Specify your query inline or reference an external SQL file, configure the database and output table with `create_table:` and `engine:` parameters, and leverage session variables like `${session_date}` to parameterize queries for dynamic data processing.
Digdag provides `_error:` blocks to define error-handling logic and `_retry:` parameters to automatically retry failed tasks with exponential backoff. Combine these patterns to build resilient workflows that recover from transient failures and trigger alerts or recovery actions when errors occur.
Yes. Design workflows with multiple `td>` tasks that extract, transform, and load data across tables using inline SQL or external scripts. Use parallel execution, conditional branching, and session variables to coordinate multi-stage pipelines, and deploy via `tdx` CLI for monitoring and scheduling.
No. Digdag uses declarative YAML syntax in `.dig` files, not code. Define tasks, schedules, and operators using straightforward configuration; SQL knowledge helps for `td>` queries, but the workflow structure itself requires only familiarity with YAML formatting and basic scheduling concepts.
Digdag provides `${session_date}`, `${session_time}`, and `${session_date_compact}` as built-in session variables that inject execution timestamps into queries. These enable time-based partitioning, incremental loads, and date-stamped outputs without manual parameter updates across workflow runs.
treasure-dataGenerate hist-union documentation with SQL examples from codebase configurations.
marquesfelipAutomate end-to-end customer offboarding workflows for SaaS platforms.
herd-labsAggregate logs, live data, and health signals from Goldsky Turbo pipelines.
minyeamerConvert YAML configuration files into executable Apache Airflow DAG definitions.