skills.rest
jerrylin96
jerrylin96COMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-dataflow

Guides authoring, packaging, launching, and diagnosing Apache Beam pipelines on Google Cloud Dataflow.

9|Updated Jul 8, 2026
Search Tags:#troubleshooting#gcp#streaming#pipeline#dataflow#apache beam#flex template
One-click install
npx skills add https://github.com/jerrylin96/dotgemini --skill gcp-dataflow-jerrylin96
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-dataflow
Source: https://github.com/jerrylin96/dotgemini/tree/main/skills/gcp-dataflow
Command: npx skills add https://github.com/jerrylin96/dotgemini --skill gcp-dataflow-jerrylin96

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Building and operating Apache Beam pipelines on Google Cloud Dataflow involves many error-prone decisions: choosing between Google-provided templates and custom code, packaging Flex Templates correctly, configuring required parameters, and diagnosing streaming job health. This Skill provides structured, opinionated workflows for each of these stages so pipelines are created, launched, and debugged correctly. ## Core Features & Use Cases - Template Discovery & Configuration: Checks the public dataflow-templates bucket for matching Google-provided Classic or Flex templates, reads template metadata, and enforces strict required-parameter validation before launch. - Pipeline Authoring & Packaging: Guides Java (Gradle), Python, and Go project setup with consistent Beam SDK versions, and packages pipelines as Flex Templates using Cloud Build with single Docker image configuration. - Diagnostics & Troubleshooting: Performs root cause analysis of streaming and batch jobs using Cloud Monitoring metrics, Cloud Logging queries, and the Dataflow REST API, covering watermarks, backlog, bottlenecks, and autoscaling. - Use Case: A user asks why their streaming pipeline is slow. The Skill maps transform names to stages, queries metrics like job/data_watermark_age and job/is_bottleneck, correlates with logs, and produces a structured diagnosis with recommendations. ## Quick Start Ask the assistant to create a new Dataflow pipeline from Pub/Sub to BigQuery, or to diagnose the performance of an existing Dataflow job by providing its job ID and project.

Frequently Asked Questions about gcp-dataflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Apache Beam pipeline on Google Dataflow?▼

First check whether a Google-provided template in gs://dataflow-templates matches your source and sink. If none matches, create a custom pipeline in Java, Python, or Go using the latest Apache Beam SDK, then package it as a Flex Template for a reproducible launch environment.

How do I build a Dataflow Flex Template with Cloud Build?▼

Provide a cloudbuild.yaml that builds and pushes your pipeline's Docker image to Artifact Registry, avoiding reliance on local Docker. For Python, use the single Docker image configuration where the launcher image also serves as the worker SDK container image.

How do I troubleshoot a slow Dataflow streaming job?▼

Query Cloud Monitoring metrics such as job/data_watermark_age, job/system_lag, job/estimated_backlog_processing_time, and job/is_bottleneck, then correlate spikes with Cloud Logging errors and Dataflow job messages. Analyze bottleneck causes, parallelism keys, and autoscaling behavior to identify root causes.

What is the difference between Dataflow Classic and Flex Templates?▼

Classic templates are pre-built jobs staged in gs://dataflow-templates with separate _metadata files, while Flex Templates package a Docker image with a template spec file for a hermetic, reproducible launch. Flex Templates are recommended when creating new custom pipelines.

Why does my Dataflow template launch fail with missing parameters?▼

Any parameter in the template metadata JSON without "isOptional": true is strictly required by the Dataflow API, even if its description suggests a default value. Identify all required parameters from the metadata file and supply every one of them at launch.

When should I not use this Dataflow skill?▼

Do not use it for general GCP resource management unrelated to Dataflow, for issues with other services like GCE or BigQuery unless they directly affect pipeline execution, or for pipeline technologies other than Apache Beam on Dataflow.

Related Skills

Explore More Software Engineering Skills→
markus41markus41
1

gcp

Automate provisioning, deployment, and orchestration of Google Cloud Platform resources via CLI, SDKs, and Terraform.

Community
Advanced
2nth-ai2nth-ai
1

tech/google/cloud/data

Query BigQuery, publish to Pub/Sub, and run Dataflow pipelines.

Official
Advanced
PeterHddPeterHdd
10

engineering-devops-automator

Automate DevOps processes with Terraform, Docker, Kubernetes, and AWS or GCP.

Community
Advanced
reason-machinesreason-machines
5

google-cloud-data-engineering-hub

Build GCP data pipelines using BigQuery, Dataflow, Beam, Composer, Pub/Sub, and Vertex AI.

Official
Advanced