gcp-spark

Run Spark ETL pipelines on Google Cloud Dataproc by orchestrating jobs that read data from BigQuery, process it with Spark, and write results to BigLake Iceberg catalogs. The workflow validates schemas before write to ensure reliable data pipelines.

Yes, you can use Spark with BigLake Iceberg catalogs and Spanner on GCP. The environment allows you to read and write data directly using these sources for data access within your Dataproc Spark workloads.

Manage interactive Spark sessions on Dataproc Serverless by coordinating workloads through an enforced Task Execution Workflow. This approach enables data engineers to deploy and debug scalable Spark jobs with reliable execution.

Yes, this workflow supports training machine learning models with Spark on GCP. You can coordinate Spark-based ML model training and run inference directly within Dataproc Clusters and Serverless environments.

Validate schemas before writing data to Iceberg tables in Spark by leveraging the enforced Task Execution Workflow. This mechanism checks data structures prior to write operations to prevent pipeline failures.

Spark jobs writing to BigQuery on Dataproc may fail if the schema validation step within the Task Execution Workflow detects a mismatch. Ensure your input data structure matches the target BigQuery table schema before execution.