skills.rest
ricardolui
ricardoluiCOMMUNITY
Explore all skills by this author →
GitHub

gcp-dataform-deployment

Deploy and orchestrate Google Cloud Dataform pipelines via REST API with IAM configuration.

Updated Jul 7, 2026
Search Tags:#deployment#data pipeline#google cloud#rest api#iam#bigquery#dataform
One-click install
npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill gcp-dataform-deployment-ricardolui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataform-deployment
Source: https://github.com/ricardolui/gcp-custom-agent-skills/tree/main/gcp-dataform-deployment
Command: npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill gcp-dataform-deployment-ricardolui

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Programmatically deploying Dataform pipelines on Google Cloud is error-prone: repositories appear hidden or "Untitled" in BigQuery Pipelines, workflow invocations fail with Act-As permission denials, and error details are hard to retrieve. This Skill provides the exact REST API patterns and IAM fixes to automate the full deployment lifecycle. ## Core Features & Use Cases - Repository & Workspace Management: Create labeled repositories with proper displayName so they appear correctly in the BigQuery Pipelines console, and manage workspaces via REST endpoints. - File Upload, Commit & Execution: Upload SQLX files with base64-encoded writeFile calls, commit changes, then trigger compilation results and workflow invocations in a two-step handshake. - IAM Troubleshooting: Resolve strict Act-As policy failures by granting Service Account Token Creator, Service Account User, and BigQuery Admin roles to the correct identities. - Use Case: A data engineer needs to deploy a customer churn SQLX pipeline from CI/CD. Use this Skill to create the repository, upload definitions, compile, invoke the workflow, and poll for failures with detailed error messages. ## Quick Start Use the gcp-dataform-deployment skill to create a Dataform repository, upload my SQLX files, and run the workflow in my GCP project.

Frequently Asked Questions about gcp-dataform-deployment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy a Dataform pipeline programmatically on Google Cloud?▼

Deploy a Dataform pipeline by creating a repository, creating a workspace, uploading SQLX files via the writeFile REST endpoint, committing the workspace, then triggering a compilation result followed by a workflow invocation. Each step uses the Dataform v1 REST API with a Bearer access token.

How to run Dataform workflow invocations via REST API?▼

Run a workflow invocation in two steps: first POST to compilationResults with the workspace path, then POST to workflowInvocations passing the returned compilation result name and an invocationConfig with the execution service account. Poll the invocation resource for RUNNING, SUCCEEDED, or FAILED states.

Why is my Dataform repository not showing in BigQuery Pipelines?▼

Repositories created programmatically are hidden from the BigQuery Pipelines UI unless they have the label bigquery-workflow set to the repository ID. The console display name also requires setting the displayName field via a PATCH request, otherwise the pipeline shows as Untitled.

Why does Dataform fail with Act-As permission denied errors?▼

Dataform fails with Act-As errors when the Dataform service agent lacks the Service Account Token Creator role on the execution service account. Grant roles/iam.serviceAccountTokenCreator to the Dataform service agent, roles/iam.serviceAccountUser to the caller, and roles/bigquery.admin to the execution account.

How do I get detailed error messages from a failed Dataform invocation?▼

Get detailed failure reasons by calling GET on the workflow invocation resource with the :query custom verb. The response lists action results where state is FAILED, and the failureReason attribute contains the direct database exception such as invalid columns or syntax errors.

Related Skills

Explore More Data & Analytics Skills→
1919114514yasenpai-maker1919114514yasenpai-maker

dataform-bigquery

Generates and validates Dataform SQLX pipeline code for BigQuery ELT workflows.

Community
Advanced
webrix-aiwebrix-ai
1

dashboard-generation

Create analytics dashboards with KPIs, charts, and automated refresh schedules.

Official
Intermediate
rose4320rose4320

gcp-pipeline-resource-provisioning

Automate GCP data pipeline resource provisioning via deployment.yaml and gcloud commands.

Community
Intermediate
Jatinkrmahato993203Jatinkrmahato993203

dbt-bigquery

Generates, modifies, and optimizes dbt models and pipelines targeting Google BigQuery.

Community
Advanced