What problem does it solve? Setting up data pipeline orchestration on Google Cloud Composer requires correctly authoring two interdependent YAML files (an orchestration pipeline definition and a deployment.yaml), matching strict schema rules, and running multiple gcloud commands in the right order. This Skill guides the agent through that entire lifecycle so pipelines are valid, deployable, and runnable. ## Core Features & Use Cases - Pipeline Generation & Updates: Creates or updates orchestration pipeline YAML and deployment.yaml files for dbt, Dataform, PySpark, notebooks, Python scripts, and inline BigQuery SQL, following the official protobuf-based schema with camelCase field mapping. - Environment Discovery & Validation: Queries Dataproc clusters and Composer environments via gcloud, verifies compatible image versions and PyPI packages, and validates files with gcloud beta orchestration-pipelines validate before completion. - Deploy & Trigger Workflow: Deploys pipelines with --local, polls for DAG registration, triggers runs, and supports pause/unpause operations, with a Python fallback script for triggering Airflow DAGs directly. - Use Case: A data engineer asks to schedule a nightly dbt run after a PySpark job. The Skill initializes the project, discovers the correct Composer environment, generates both YAML files with proper schedules and dependencies, validates them, deploys, and triggers the first run. ## Quick Start Ask the agent to create and deploy a Cloud Composer orchestration pipeline for your dbt project or Spark job in a given GCP project and region.
