skills.rest
jerrylin96
jerrylin96COMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-pipeline-resource-provisioning

Automates declarative provisioning of BigQuery, Dataform, and Dataproc resources via deployment.yaml.

9|Updated Jul 8, 2026
Search Tags:#gcp#data pipeline#bigquery#provisioning#dataform#dataproc#deployment.yaml
One-click install
npx skills add https://github.com/jerrylin96/dotgemini --skill gcp-pipeline-resource-provisioning-jerrylin96
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-pipeline-resource-provisioning
Source: https://github.com/jerrylin96/dotgemini/tree/main/skills/gcp-pipeline-resource-provisioning
Command: npx skills add https://github.com/jerrylin96/dotgemini --skill gcp-pipeline-resource-provisioning-jerrylin96

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Manually creating and configuring Google Cloud data pipeline resources across dev, staging, and prod environments is error-prone and inconsistent. This Skill automates declarative resource provisioning through a single deployment.yaml file, ensuring environment-specific configurations are validated and deployed correctly. ## Core Features & Use Cases - Declarative Resource Provisioning: Define BigQuery datasets/tables, Dataform repositories, Dataproc clusters, and BigQuery Data Transfer Service configs in one deployment.yaml file. - Environment Management: Map dev, staging, and prod environments to their own projects, regions, variables, and secrets with automatic variable substitution. - Secure Secret Handling: Reference existing Secret Manager secrets without ever reading or exposing their values during deployment. - Use Case: A data engineer needs to set up a new BigQuery dataset with a DTS transfer config and a Dataform workflow for the staging environment. The Skill discovers project parameters, generates the deployment.yaml, validates it, and deploys all resources with one gcloud command. ## Quick Start Ask the agent to create a deployment.yaml that provisions a BigQuery dataset and table in the dev environment, then validate and deploy it.

Frequently Asked Questions about gcp-pipeline-resource-provisioning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I provision BigQuery datasets and tables with deployment.yaml?▼

Define bigquery.dataset and bigquery.table resources under the target environment in deployment.yaml, including the mandatory datacloud label. Then run gcloud beta orchestration-pipelines validate followed by deploy with the --environment flag.

What GCP resource types can deployment.yaml provision?▼

Run gcloud beta orchestration-pipelines resource-types list to see all supported types. Common examples include BigQuery datasets and tables, Dataform repositories and workflow configs, Dataproc clusters and workflow templates, and DTS transfer configs.

How do I handle secrets in deployment.yaml securely?▼

Declare secrets in the secrets block referencing existing Secret Manager paths, then substitute them with {{ SECRET_NAME }} in resource definitions. Never hardcode plain-text secrets, and create missing secrets manually since the framework only references existing ones.

When should I use Terraform instead of deployment.yaml provisioning?▼

Use Terraform for general cloud infrastructure like VMs, networks, Kubernetes, and IAM policies, or when infrastructure spans multiple cloud providers. This approach targets supported data pipeline resources only and should not manage resources already under Terraform control.

Why does deployment.yaml validation fail before deployment?▼

Validation fails when placeholders like YOUR_PROJECT_ID are not replaced with discovered values, variables are unresolvable, or the YAML syntax is incorrect. Run gcloud config get project and related discovery commands to populate actual values first.

Related Skills

Explore More Data & Analytics Skills→
BagelHoleBagelHole
46

gcp-secret-manager

Manage Google Cloud Secret Manager secrets with IAM and GKE integration.

Community
Intermediate
ironkid90ironkid90

bigquery-data-transfer-service

Discover and manage BigQuery Data Transfer Service configurations via the Data Transfer API.

Community
Intermediate
AgentientAgentient
2

infrastructure-as-code

Provision GCP infrastructure with Terraform using modular patterns and remote state.

Official
Advanced
All-The-VibesAll-The-Vibes
4

infrastructure

Provision and manage cloud resources using declarative infrastructure as code tooling.

Official
Advanced