gcp-data-pipelines

Choose dbt for SQL-first teams with modular models and built-in tests running inside BigQuery, Dataflow for streaming and unified batch processing with Apache Beam, and Dataform for Google-native ELT with SQLX and GCP Console integration. The Skill presents these options and highlights the most likely fit based on your request.

It scans for indicator files: dbt_project.yml for dbt, workflow_settings.yaml or dataform.json for Dataform, Apache Beam imports for Dataflow, PySpark imports for Spark, and deployment.yaml for orchestration. Unambiguous indicators route directly to the matching skill.

Use Cloud Composer when you need to schedule, deploy, or coordinate multiple existing pipelines such as dbt and Spark as a unified workflow. Requests mentioning scheduling, cron, or automation are routed to the gcp-pipeline-orchestration skill.

Running directly executes the pipeline in your development environment using commands like dbt run or gcloud dataproc jobs submit. Orchestrating deploys the pipeline to a managed service like Cloud Composer and triggers it as part of a larger scheduled workflow.

Not necessarily. A .py file could be Spark, Airflow, or generic Python. The Skill confirms with you which pipeline type you are working with unless it finds an explicit indicator like a PySpark import.