skills.rest
jerrylin96
jerrylin96COMMUNITY
Explore all skills by this author →
WebsiteGitHub

gcp-spark

Generates and executes PySpark code on Google Cloud Dataproc clusters and Serverless.

9|Updated Jul 8, 2026
Search Tags:#gcp#etl#spark#bigquery#pyspark#iceberg#dataproc
One-click install
npx skills add https://github.com/jerrylin96/dotgemini --skill gcp-spark-jerrylin96
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/jerrylin96/dotgemini/tree/main/skills/gcp-spark
Command: npx skills add https://github.com/jerrylin96/dotgemini --skill gcp-spark-jerrylin96

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Writing correct Spark code for Google Cloud requires knowing Dataproc submission commands, BigLake Iceberg catalog configuration, BigQuery and Spanner connector options, and ML library constraints. This Skill encodes those verified patterns so generated PySpark notebooks and scripts run correctly the first time. ## Core Features & Use Cases - Guided Spark Code Generation: Produces Python notebooks (.ipynb) following a mandatory workflow: schema discovery, code generation, schema verification, and compilation checks before execution. - Data Source Integration: Provides verified read/write patterns for BigQuery, BigLake Iceberg catalogs (GCS and S3 storage), Google Cloud Storage, and Spanner. - Dataproc Resource Management: Covers cluster and job listing, Serverless batch submission with Iceberg/Spanner/XGBoost properties, and interactive session guidance via gcloud or MCP tools. - Use Case: Ask for a Spark ETL pipeline that reads a CSV from GCS, enriches it with a BigQuery reference table, and writes results to a BigLake Iceberg table, and receive a validated notebook with correct catalog configuration. ## Quick Start Ask the agent to write a PySpark notebook that reads a BigQuery table and writes the results to a BigLake Iceberg table on Dataproc Serverless.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I submit a PySpark job to Dataproc Serverless?▼

Use gcloud dataproc batches submit pyspark with your script path, project, region, version 2.3, and a --deps-bucket pointing to a GCS path for dependencies. Expect initial batch execution to take 10-15 minutes, which is normal behavior.

How do I read and write BigLake Iceberg tables from Spark?▼

Configure a SparkCatalog with the BigLake REST endpoint, GoogleAuthManager, and the correct io-impl (GCSFileIO or S3FileIO) based on the table's metadataPath. Set the current catalog, then read with spark.read.format("iceberg") and write using the DataFrameWriterV2 writeTo API.

Can I use LightGBM with Spark on Dataproc Serverless?▼

LightGBM has dependency conflicts on Dataproc Serverless because the SynapseML wrapper clashes with Dataproc's internal libraries. Use XGBoost, native Spark ML like GBTClassifier, or train on a dedicated Dataproc cluster instead.

Why does SparkXGBClassifier fail when I set the objective parameter?▼

SparkXGBClassifier does not allow a custom objective parameter; it is inferred automatically from the classifier type. Setting objective raises a ValueError, and you must also disable dynamic allocation for XGBoost workloads.

When should I not use this Spark skill?▼

Avoid it for generic Python scripts that do not use Spark and for simple SQL queries that can run directly in BigQuery. It is designed for Spark ETL, ML workloads, and Dataproc resource management on GCP.

Related Skills

Explore More Data & Analytics Skills→
arslan9024arslan9024

gcp-spark

Generates and executes Spark code on Google Cloud Dataproc clusters and Serverless batches.

Community
Advanced
JeffallanJeffallan
10.9k

spark-engineer

Diagnose and resolve performance bottlenecks in Spark data pipelines.

Community
Advanced
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced
2nth-ai2nth-ai
1

tech/google/cloud/data

Query BigQuery, publish to Pub/Sub, and run Dataflow pipelines.

Official
Advanced