skills.rest
AubreyHan
AubreyHanCOMMUNITY
Explore all skills by this author →
GitHub

gcp-pipeline-orchestration

Generate and update Google Cloud Composer orchestration YAML for data pipelines.

Updated Jun 10, 2026
Search Tags:#deployment#validation#gcp#orchestration#composer#airflow#data-pipelines
One-click install
npx skills add https://github.com/AubreyHan/SKILL_Repo --skill gcp-pipeline-orchestration-aubreyhan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-pipeline-orchestration
Source: https://github.com/AubreyHan/SKILL_Repo/tree/main/gcp-pipeline-orchestration
Command: npx skills add https://github.com/AubreyHan/SKILL_Repo --skill gcp-pipeline-orchestration-aubreyhan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires google-auth, requests, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill removes the manual burden of building and maintaining Google Cloud Composer orchestration files for data workflows, so teams can create, update, deploy, and trigger pipelines consistently.

Core Features & Use Cases

  • Pipeline Definition Management: Generate or update orchestration YAML for dbt, notebooks, Spark jobs, Dataform, Python scripts, and BigQuery SQL.
  • Deployment Readiness: Keep orchestration and deployment files aligned, enforce required schedule fields, and validate Composer compatibility before release.
  • Operational Control: Deploy, trigger, pause, or resume pipelines in the correct Composer environment with safer configuration handling.
  • Use Case: A data engineer can turn a mixed dbt-and-PySpark workflow into a deployable Composer pipeline and validate it against the target environment.

Quick Start

Ask the assistant to create or update a Google Cloud Composer orchestration pipeline for my project and validate it for the target environment.

Frequently Asked Questions about gcp-pipeline-orchestration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate dbt and PySpark workflows in Google Cloud Composer?▼

To orchestrate dbt and PySpark workflows in Google Cloud Composer, you can generate and update orchestration YAML definitions that integrate these mixed data jobs into a single deployable pipeline.

What is the best way to manage BigQuery SQL and Dataform pipeline deployments on GCP?▼

Managing BigQuery SQL and Dataform pipeline deployments on GCP involves generating camelCase YAML orchestration files, keeping deployment configurations aligned, and triggering pipelines within the correct Composer environment.

Does Google Cloud Composer support scheduling Python scripts and notebooks?▼

Google Cloud Composer supports scheduling Python scripts and notebooks by generating orchestration definitions that bundle these components into end-to-end data pipelines with operational controls to pause, resume, or trigger executions.

Why do my Composer pipeline deployments fail schedule validation?▼

Composer pipeline deployments fail schedule validation when required schedule fields, such as endTime, are missing or misconfigured in your mandatory orchestration and deployment YAML files.

Can I update existing GCP data pipelines without rewriting the entire orchestration configuration?▼

You can update existing GCP data pipelines by modifying the orchestration YAML definitions directly, allowing you to add new dbt models, Spark jobs, or BigQuery SQL tasks while maintaining Composer environment compatibility.

Related Skills

Explore More Software Engineering Skills→
ilorozco11ilorozco11

composer

Provision and manage Google Cloud Composer environments with Terraform.

Community
Advanced
rory-datarory-data

testing-dags

Trigger Airflow DAG runs, diagnose failures, inspect logs, and apply fixes in iterative cycles.

Community
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-pipeline-orchestration

Automate Cloud Composer pipeline creation and management with gcloud validation.

Official
Advanced
AubreyHanAubreyHan

gcp-composer-troubleshooting

Diagnose failed Cloud Composer and Airflow pipelines via Cloud Logging and DAG retrieval.

Community
Advanced