skills.rest
AubreyHan
AubreyHanCOMMUNITY
Explore all skills by this author →
GitHub

gcp-spark

Build and validate Spark workflows for Google Cloud Dataproc.

Updated Jun 10, 2026
Search Tags:#gcp#spark#bigquery#iceberg#notebook#spanner#dataproc
One-click install
npx skills add https://github.com/AubreyHan/SKILL_Repo --skill gcp-spark-aubreyhan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/AubreyHan/SKILL_Repo/tree/main/gcp-spark
Command: npx skills add https://github.com/AubreyHan/SKILL_Repo --skill gcp-spark-aubreyhan

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill removes the guesswork from building, running, and debugging Spark workloads on Google Cloud Dataproc so data teams can move from source to validated output with fewer failures and fewer manual checks.

Core Features & Use Cases

  • Dataproc Workflow Guidance: Supports Spark development for clusters, serverless batches, and interactive sessions with clear execution rules.
  • GCP Data Integration: Reads and writes data across BigQuery, BigLake Iceberg catalogs, Google Cloud Storage, and Spanner.
  • Validation and Optimization: Emphasizes schema discovery, schema verification before writes, and Spark optimization practices to reduce runtime issues.
  • ML on Spark: Covers Spark-based machine learning tasks such as training and inference with supported patterns and platform constraints.
  • Use Case: A data engineer can turn a raw dataset into a production-ready Spark notebook that validates schemas, applies the right connector configuration, and writes results safely to the target GCP system.

Quick Start

Use the gcp-spark skill to generate a Dataproc-ready Spark notebook that reads the source data, verifies schemas, applies the required optimizations, and writes the result to the specified GCP destination.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a Spark workflow for Google Cloud Dataproc that writes to BigQuery?▼

To build a Spark workflow for Dataproc, you need a notebook-first output that performs schema discovery and verification before writing. The workflow applies correct connector configurations and IAM handling to safely write results to BigQuery.

Can I use Spark with BigLake Iceberg catalogs on GCP?▼

Yes, Spark works with BigLake Iceberg catalogs on GCP. Building these workflows requires correct connector configuration and IAM handling to read and write data safely across Google Cloud Storage and Iceberg tables.

What's the best way to validate Spark schemas before writing to Spanner?▼

The best way to validate Spark schemas before writing to Spanner is to enforce schema discovery and schema verification steps prior to writes. This process reduces runtime failures and ensures data matches the target GCP system.

Does Dataproc serverless support interactive Spark sessions?▼

Yes, Dataproc supports interactive Spark sessions alongside serverless batches and cluster-based execution. Building workflows for these environments requires applying specific execution rules and Spark optimization practices.

How do I optimize Spark machine learning jobs on GCP?▼

To optimize Spark machine learning jobs on GCP, you apply supported training and inference patterns alongside platform constraints. Following Spark optimization practices during workflow development reduces runtime issues.

Related Skills

Explore More Data & Analytics Skills→
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced
ironkid90ironkid90

gcp-spark

Automates Spark code execution on Google Cloud Dataproc with BigLake, BigQuery, and Spanner.

Community
Advanced
DicklesworthstoneDicklesworthstone
70

gcloud

Manage Google Cloud resources via the gcloud CLI.

Community
Advanced
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced