transform

Execute named data transformation tasks in Starlake pipelines with YAML definitions.

209|30|Updated Sep 18, 2021

Search Tags:#etl #data-pipeline #task-runner #transform #starlake #yaml-based

Ready to run

npx skills add https://github.com/starlake-ai/starlake --skill transform-starlake-ai

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: transform
Source: https://github.com/starlake-ai/starlake/tree/main/.agent/skills/transform
Command: npx skills add https://github.com/starlake-ai/starlake --skill transform-starlake-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Transform tasks in data pipelines often require manual scripting or ad-hoc SQL work. This skill provides a standard, declarative way to run predefined transformation steps within Starlake without writing custom code each time.

Core Features & Use Cases

Declarative task execution: Run named transformation tasks with a single command and optional parameters.
Flexible options: Supports --name, --compile, --sync-apply, --sync-preview, --query, --dry-run, --interactive, --reload, --truncate, --recursive, --test, --options, and --scheduledDate to control execution and output.
Use Case: Automate routine data transformations across ETL pipelines, ensure consistent transformations across environments, and validate SQL generation before execution.

Quick Start

Run a transformation task by specifying its name, for example: starlake transform --name mydomain.mytask

Frequently Asked Questions about transform

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run a data transformation task in an ETL pipeline without writing custom code?▼

You can run data transformation tasks declaratively by executing a single command with the task name. This requires YAML-based task definitions integrated with the Starlake transform workflow to parse options and execute the specified task.

Can I validate SQL generation before executing a transformation in a data pipeline?▼

Yes, you can validate SQL generation before execution using the dry-run option. This allows you to preview the transformation output and ensure SQL accuracy within your Starlake pipeline without applying actual changes.

How does a YAML-based task runner handle recursive dependencies in ETL pipelines?▼

A YAML-based task runner handles recursive dependencies by parsing task definitions to identify and execute prerequisite transformations in the correct order. This ensures all dependent tasks are resolved before the target task runs.

What options are available for overriding queries when running ETL data transformations?▼

You can override queries during ETL data transformations by using the --query option. This allows you to specify custom SQL directly, replacing the default query defined in your YAML-based task configuration.

Do I need to define task names to automate routine data transformations across environments?▼

Yes, you must define task names to automate routine data transformations. Using the --name option with a structured identifier like mydomain.mytask ensures consistent execution across different environments.

What is the best way to execute scheduled data transformations with specific execution dates?▼

The best way to execute scheduled data transformations is by using the --scheduledDate option. This parameter allows you to specify a particular execution date, ensuring the transformation task processes the correct data partition.