gcp-spark

Execute and manage Apache Spark workloads on Google Cloud Dataproc clusters.

2|1|Updated May 25, 2026

Search Tags:#gcp #etl #spark #bigquery #iceberg #dataproc

Ready to run

npx skills add https://github.com/interflownex/All-in-One --skill gcp-spark-interflownex

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: gcp-spark
Source: https://github.com/interflownex/All-in-One/tree/main/.gemini/skills/gcp-spark
Command: npx skills add https://github.com/interflownex/All-in-One --skill gcp-spark-interflownex

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill streamlines the development, execution, and optimization of Apache Spark workloads on Google Cloud Platform, removing the complexity of managing Dataproc clusters and serverless configurations.

Core Features & Use Cases

Workload Management: Seamlessly handle Spark ETL pipelines, ML training, and interactive sessions on Dataproc.
Data Integration: Native support for reading and writing data across BigQuery, BigLake Iceberg, and Spanner.
Performance Optimization: Built-in guidance for broadcast joins, schema validation, and resource management to prevent common failures like out-of-memory errors.

Quick Start

Use the gcp-spark skill to submit a pyspark script named etl_job.py to a Dataproc serverless batch in the us-central1 region.

Frequently Asked Questions about gcp-spark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I submit a PySpark script to a Dataproc serverless batch?▼

To run a PySpark script on Dataproc serverless, you configure the batch workload parameters and target region, such as us-central1. This approach executes Spark ETL pipelines without requiring manual cluster provisioning or management.

Can Spark on Dataproc read and write data directly to BigQuery and BigLake Iceberg?▼

Yes, Spark on Dataproc supports native data integration for reading and writing directly across BigQuery, BigLake Iceberg, and Spanner. This capability streamlines ETL pipeline development and data integration workflows within Google Cloud Platform.

What is the best way to prevent out-of-memory errors in Spark ETL pipelines on GCP?▼

The best way to prevent out-of-memory errors in Spark ETL pipelines on GCP is to apply built-in performance optimization patterns like broadcast joins, schema validation, and resource management. These practices ensure production-grade stability for Dataproc workloads.

Does this approach support machine learning model training on Dataproc?▼

Yes, executing and managing Apache Spark workloads on Google Cloud Dataproc clusters and serverless environments fully supports machine learning model training. It provides the necessary infrastructure and resource optimization patterns for ML development.

When should I use Dataproc serverless instead of managing my own Spark clusters?▼

You should use Dataproc serverless when you need to streamline workload execution and remove the complexity of managing Dataproc clusters. It is ideal for running interactive sessions and ETL pipelines with built-in resource optimization for production-grade stability.