gcp-spark

This Skill automates Spark code execution on Dataproc Clusters and Serverless, leveraging BigLake Iceberg and BigQuery integration to read and write data for ETL pipelines and ML workflows on GCP.

Running Spark ETL pipelines on GCP requires existing Dataproc and BigLake setup. You must configure your Dataproc clusters or serverless environment before executing automated Spark code for data processing.

Yes, this Skill supports using Spark with BigLake Iceberg catalogs, BigQuery, and Spanner for reading and writing data, enabling comprehensive data processing across multiple GCP storage systems for ETL and ML workflows.

To debug Spark execution failures on Dataproc, this Skill provides debugging capabilities to identify and resolve errors during automated Spark job execution on GCP clusters and serverless environments.

Yes, this Skill supports training and running ML models with Spark on GCP Dataproc. It leverages BigLake Iceberg and BigQuery integration to handle the data processing required for ML workflows.

You should avoid using Dataproc for Spark data processing if you lack the required Dataproc and BigLake setup, or if your workflows do not require BigQuery, BigLake Iceberg, or Spanner integration for ETL and ML tasks.