skills.rest
nguyenhungtran18
nguyenhungtran18COMMUNITY
Explore all skills by this author →
GitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

1|Updated Aug 31, 2026
Search Tags:#gcp#DataFlow#streaming#autoscaling#apache beam#pipeline diagnostics#flex template
One-click install
npx skills add https://github.com/nguyenhungtran18/skill-and-tool-tracker --skill gcp-dataflow-nguyenhungtran18
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/nguyenhungtran18/skill-and-tool-tracker/tree/main/skills/gcp_dataflow
Command: npx skills add https://github.com/nguyenhungtran18/skill-and-tool-tracker --skill gcp-dataflow-nguyenhungtran18

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates correctly, validating required parameters, and diagnosing streaming job health, bottlenecks, and autoscaling behavior. This Skill encodes those workflows so pipelines are created, launched, and troubleshot correctly the first time. ## Core Features & Use Cases - Template Discovery & Configuration: Checks the public dataflow-templates bucket for matching Google-provided Classic or Flex templates, reads template metadata, and enforces strict required-parameter validation before launch. - Pipeline Authoring & Packaging: Sets up Java (Gradle), Python, or Go Beam projects with consistent SDK versions, and packages pipelines as Flex Templates using Cloud Build with single Docker image configuration. - Diagnostics & Troubleshooting: Correlates Cloud Monitoring metrics, Cloud Logging queries, and Dataflow REST API job messages to analyze streaming job health, bottlenecks, parallelism, and autoscaling anomalies. - Use Case: A data engineer needs to move data from Pub/Sub to BigQuery. The Skill first checks for an existing Google template, pre-creates the target and dead-letter tables, drafts the full launch command for approval, then monitors the job and diagnoses any watermark lag or hot-key bottlenecks. ## Quick Start Use the gcp-dataflow skill to create an Apache Beam pipeline that streams data from Pub/Sub to BigQuery and package it as a Flex Template.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Apache Beam pipeline on Google Dataflow?▼

First check the public dataflow-templates bucket for a Google-provided template matching your source and sink. If none fits, create a custom pipeline in Java, Python, or Go using the latest Beam SDK version, and package it as a Flex Template for a reproducible launch environment.

How do I build a Dataflow Flex Template with Cloud Build?▼

Provide a cloudbuild.yaml that builds and pushes the Docker image, then register the template spec in GCS. For Python, use the single Docker image configuration where the launcher and worker runtime share one image, and pass it via the sdk_container_image parameter.

How do I troubleshoot a slow Dataflow streaming job?▼

Correlate job messages from the Dataflow REST API with Monitoring metrics like data_watermark_age, is_bottleneck, and backlogged_keys, plus worker logs from Cloud Logging. Classify job health by watermark delay, then investigate bottlenecks, hot keys, or stuck operations.

Why is my Dataflow autoscaling adding workers but backlog keeps growing?▼

This indicates undetected throttling: IO latency, locks, or hot keys leave workers idle with low CPU while backlog rises, so the autoscaler scales to the maximum limit. Mitigate with in-flight updates to worker utilization hints or min/max worker clamps.

Does Dataflow support secrets in Flex Template workers?▼

Yes, but never bake certificates or keys into Docker image layers. Retrieve them at runtime inside DoFn.setup() using the Secret Manager client library, and grant the worker service account the secretmanager.secretAccessor role.

When should I not use this Dataflow skill?▼

Do not use it for general GCP resource management unrelated to Dataflow, issues with other services like GCE or BigQuery unless they directly impact pipeline execution, or pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
markus41markus41
1

gcp

Automate provisioning, deployment, and orchestration of Google Cloud Platform resources via CLI, SDKs, and Terraform.

Community
Advanced
reason-machinesreason-machines
5

google-cloud-data-engineering-hub

Build GCP data pipelines using BigQuery, Dataflow, Beam, Composer, Pub/Sub, and Vertex AI.

Official
Advanced
way-platformway-platform
3

spacelift-terraform

Manage Spacelift stacks, contexts, policies, and worker pools via Terraform GitOps workflows.

Official
Advanced
SmileAfterBurnSmileAfterBurn

gcp-data-pipelines

Routes data pipeline requests to the appropriate Google Cloud tool skill.

Community
Intermediate