skills.rest
ricardolui
ricardoluiCOMMUNITY
Explore all skills by this author →
GitHub

federate-lakehouse-catalog

Configures Google Cloud Lakehouse federated catalogs connecting to Databricks Unity and AWS Glue.

Updated Jul 7, 2026
Search Tags:#bigquery#iceberg#lakehouse#cross-cloud#aws glue#federated catalog#databricks unity
One-click install
npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill federate-lakehouse-catalog-ricardolui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: federate-lakehouse-catalog
Source: https://github.com/ricardolui/gcp-custom-agent-skills/tree/main/federate-lakehouse-catalog
Command: npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill federate-lakehouse-catalog-ricardolui

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Querying data stored in remote catalogs like Databricks Unity or AWS Glue from Google Cloud engines normally requires complex cross-cloud setup involving credentials, IAM roles, and region alignment. This Skill guides you through creating a federated BigLake Iceberg catalog so you can query remote data directly from BigQuery or Spark on GCP. ## Core Features & Use Cases - Databricks Unity Federation: Stores OAuth credentials in regional Secret Manager and creates a federated catalog pointing to a Unity Catalog instance. - AWS Glue Federation: Walks through IAM role creation with placeholder trust policies, catalog creation, and trust policy updates using the Google service account ID. - Region Pairing Guidance: Recommends optimal GCP regions paired with AWS regions to minimize cross-cloud latency. - Use Case: A data team keeps tables in AWS Glue on S3 but wants analysts to query them from BigQuery without migrating data; this Skill sets up the federation end to end. ## Quick Start Ask the assistant to federate your lakehouse catalog to Databricks Unity or AWS Glue so you can query that data from BigQuery.

Frequently Asked Questions about federate-lakehouse-catalog

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I query Databricks Unity Catalog data from BigQuery?▼

Create a federated BigLake Iceberg catalog of type unity using gcloud alpha biglake iceberg catalogs create, storing your Databricks OAuth client ID and secret in a regional Secret Manager secret first. Once created, query tables with standard BigQuery SQL referencing the catalog name.

How to connect AWS Glue Data Catalog to Google Cloud BigQuery?▼

Create an AWS IAM role with a placeholder trust policy, attach Glue and S3 read permissions, then create a federated BigLake catalog with the glue-warehouse set to your AWS account ID. Finally update the trust policy with the catalog's Google service account ID and enable background refresh.

Which GCP region should I pair with my AWS region for federation?▼

Choose GCP regions with low-latency dedicated connections to your AWS region, such as AWS us-east-1 with GCP us-east4, or AWS eu-west-2 with GCP europe-west2. Matching regions minimizes latency when federating large datasets across clouds.

Why does my federated catalog fail to list namespaces after creation?▼

Common causes include the AWS trust policy not referencing the correct biglake-service-account-id, the Secret Manager secret being in a different region than the catalog, or missing secretAccessor permissions. Verify region alignment and IAM bindings, then consult the cross-cloud lakehouse troubleshooting documentation.

Can this skill run Databricks SQL queries or manage AWS Glue jobs?▼

No. This skill only sets up federated catalog connections so GCP engines can query remote metadata and data. It does not execute SQL directly on Databricks, manage Databricks clusters, or administer AWS Glue jobs.

Related Skills

Explore More Data & Analytics Skills→
aaronachermannaaronachermann

databricks-lakebase-provisioned

Provision Lakebase Provisioned PostgreSQL instances and integrate Databricks Apps with OAuth authentication.

Community
Advanced
YashyasikYashyasik

federate-lakehouse-catalog

Configures Google Cloud Lakehouse federated catalogs connecting to Databricks Unity or AWS Glue.

Community
Advanced
tottenjordantottenjordan
1

federate-lakehouse-catalog

Configures Google Cloud Lakehouse federated catalogs connecting BigQuery to Databricks Unity and AWS Glue.

Community
Advanced
Overdrive-ConsultingOverdrive-Consulting
3

export-csv

Convert pipeline JSON files into CSV format with flattened nested objects.

Official
Intermediate