skills.rest
ricardolui
ricardoluiCOMMUNITY
Explore all skills by this author →
GitHub

google-iceberg-spark-bigquery

Configures a Google Cloud Lakehouse integrating Apache Iceberg, Dataproc Spark, and BigQuery.

Updated Jul 7, 2026
Search Tags:#google cloud#bigquery#pyspark#lakehouse#dataproc#apache iceberg#credential vending
One-click install
npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill google-iceberg-spark-bigquery-ricardolui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: google-iceberg-spark-bigquery
Source: https://github.com/ricardolui/gcp-custom-agent-skills/tree/main/google-iceberg-spark-bigquery
Command: npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill google-iceberg-spark-bigquery-ricardolui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyspark, google-cloud-dataproc-spark-connect.

What problem does it solve? Setting up an open-format Lakehouse on Google Cloud requires coordinating Apache Iceberg catalogs, credential vending, Spark session configuration, and BigQuery interoperability, which involves many error-prone manual steps across gcloud, PySpark, and SQL. ## Core Features & Use Cases - Lakehouse Catalog Setup: Create BigLake Iceberg REST catalogs with vended-credentials and grant the catalog service account access to GCS buckets. - Spark Configuration Patterns: Configure Dataproc Serverless batches, standard PySpark sessions, and Spark Connect sessions for BigQuery Studio or Colab Enterprise notebooks. - Cross-Engine Interoperability: Enable BigQuery DML and table management preview features so Spark and BigQuery can read and write the same Iceberg tables with ACID transactions. - Use Case: A data engineer creates an Iceberg table via a Dataproc Serverless PySpark job, then runs UPDATE statements on the same table directly from BigQuery without copying data. ## Quick Start Ask the assistant to create a Lakehouse Iceberg REST catalog with credential vending and configure a Dataproc Serverless PySpark job to write a table that BigQuery can query.

Frequently Asked Questions about google-iceberg-spark-bigquery

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Iceberg REST catalog on Google Cloud?▼

Use gcloud biglake iceberg catalogs create with --catalog-type=biglake, a default GCS location, and --credential-mode=vended-credentials. Then grant the catalog's auto-provisioned service account the Storage Object User role on your bucket.

How to connect PySpark to BigLake Iceberg catalog?▼

Configure SparkSession with spark.sql.catalog properties pointing to https://biglake.googleapis.com/iceberg/v1/restcatalog, using GoogleAuthManager for auth and GCSFileIO for storage. For Dataproc Serverless, use the dataproc.lakehouse.catalog property instead.

Can BigQuery and Spark write to the same Iceberg table?▼

Yes, when the table is created with 'gcp.biglake.bigquery-dml.enabled' and 'gcp.biglake.table-management.enabled' table properties. Both engines can then run ACID DML operations against the same Iceberg table registered in the Lakehouse catalog.

What is credential vending in BigLake Iceberg catalogs?▼

Credential vending is an access delegation mode where the catalog service account issues short-lived GCS access tokens to clients. Client applications like Spark do not need direct IAM permissions on the storage bucket.

How do I configure Spark Connect in BigQuery Studio for Iceberg?▼

Use the DataprocSparkSession builder from google.cloud.dataproc_spark_connect with a Session object whose runtime_config.properties contain the Iceberg REST catalog settings. This differs from standard SparkSession configuration used in batch jobs.

Why does my Spark job fail to access GCS through the Iceberg catalog?▼

The catalog's service account likely lacks permissions on the bucket. Retrieve it with gcloud biglake iceberg catalogs describe and grant roles/storage.objectUser on the GCS bucket holding your Iceberg data.

Related Skills

Explore More Data & Analytics Skills→
dousudousu
3

cloud-integration-tests

Execute integration tests for GCP and AWS cloud storage operations.

Community
Advanced
supercent-iosupercent-io

looker-studio-bigquery

Connect BigQuery data sources to Looker Studio dashboards.

Official
Intermediate
gemini-cli-extensionsgemini-cli-extensions
161

gcp-spark

Coordinate Spark workloads on Google Cloud Dataproc and Serverless.

Official
Advanced
mkgs-databricks-demosmkgs-databricks-demos

databricks-iceberg

Manage and access Apache Iceberg tables within Databricks.

Official
Advanced