skills.rest
jerrylin96
jerrylin96COMMUNITY
Explore all skills by this author →
WebsiteGitHub

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for data ingestion pipelines.

9|Updated Jul 8, 2026
Search Tags:#gcp#data pipeline#data ingestion#etl#gcloud#bigquery#data transfer service
One-click install
npx skills add https://github.com/jerrylin96/dotgemini --skill bigquery-data-transfer-service-jerrylin96
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bigquery-data-transfer-service
Source: https://github.com/jerrylin96/dotgemini/tree/main/skills/bigquery-data-transfer-service
Command: npx skills add https://github.com/jerrylin96/dotgemini --skill bigquery-data-transfer-service-jerrylin96

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? When building data pipelines on Google Cloud, teams often don't know whether ingestion for a data source is already managed by an existing BigQuery Data Transfer Service (DTS) transfer, or what parameters a new transfer requires. This Skill discovers existing transfer configs, extracts their metadata, and guides the declarative provisioning of new transfers. ## Core Features & Use Cases - Transfer Discovery: Lists and inspects existing DTS transfer configs in a project and region using bq CLI commands, including checking for successful runs and disabled transfers. - Data Source Parameter Discovery: Runs a Python script against the BigQuery Data Transfer REST API to enumerate available data source IDs and retrieve required parameters, including OAuth authorization URL generation for Google data sources. - Declarative Provisioning Enforcement: Ensures all new transfers are created through a deployment.yaml resource provisioning framework rather than imperative CLI commands, with mandatory user confirmation of parameters. - Use Case: A user asks to ingest Google Ads data into BigQuery. The Skill checks for existing transfers, discovers the required data source parameters, walks the user through OAuth authorization, and verifies a successful transfer run before downstream pipeline work continues. ## Quick Start Ask the assistant to check whether a BigQuery Data Transfer Service transfer already exists for your data source in your GCP project and region before creating a new ingestion pipeline.

Frequently Asked Questions about bigquery-data-transfer-service

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check for existing BigQuery Data Transfer Service configs?▼

Run bq ls --transfer_config with the target transfer location and project ID to list existing transfers. Then use bq show --format=prettyjson --transfer_config with the resource name to extract full configuration details.

How do I find required parameters for a BigQuery DTS data source?▼

Call the BigQuery Data Transfer REST API dataSources endpoint for your project and region, or run the provided Python discovery script with the data source ID. It returns all required parameters and, for Google data sources, generates an OAuth authorization URL.

Can I create BigQuery DTS transfers with bq mk or gcloud commands?▼

No, new transfer configurations must be provisioned declaratively through the gcp pipeline resource provisioning framework with a generated deployment.yaml. CLI commands are permitted only for discovery, listing, and triggering manual transfer runs.

How do I trigger a manual BigQuery DTS transfer run?▼

Use bq mk --transfer_run with the transfer config resource name and a run_time in UTC ISO format. Then poll run status every 30-60 seconds for up to 5 minutes using bq ls --transfer_run.

What should I do if a BigQuery DTS transfer run takes too long?▼

If the run is still in progress after 5 minutes of polling, stop and ask the user for proceed guidance once ingestion finishes. Do not continue building downstream pipeline tasks until the transfer completes or the user confirms.

Related Skills

Explore More Data & Analytics Skills→
kubiyabotkubiyabot
16

gcp

Manage Google Cloud resources by wrapping the native gcloud CLI.

Official
Advanced
tottenjordantottenjordan
1

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

Community
Advanced
FeexSystemsFeexSystems

gcp-pipeline-resource-provisioning

Provisions BigQuery, Dataform, and Dataproc resources declaratively via deployment.yaml.

Community
Intermediate
tottenjordantottenjordan
1

bigquery-data-transfer-service

Discovers and inspects BigQuery Data Transfer Service configurations for ingestion pipelines.

Community
Advanced