skills.rest
ricardolui
ricardoluiCOMMUNITY
Explore all skills by this author →
GitHub

databricks-bq-parity-checker

Validates hourly data parity between BigQuery Iceberg tables and Databricks Unity Catalog tables.

Updated Jul 7, 2026
Search Tags:#data validation#sql#reconciliation#bigquery#databricks#unity catalog#data parity
One-click install
npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill databricks-bq-parity-checker-ricardolui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-bq-parity-checker
Source: https://github.com/ricardolui/gcp-custom-agent-skills/tree/main/databricks-bq-parity-checker
Command: npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill databricks-bq-parity-checker-ricardolui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-sdk, and includes scripts (resource) components.

What problem does it solve? Reconciling row counts between Google Cloud BigQuery (_kfkevh Bronze/Silver tables) and Azure Databricks Unity Catalog is error-prone because the two platforms use different timestamp semantics (broker enqueue time vs. payload StorageDate), causing false mismatches at hour boundaries and costly unpartitioned scans. ## Core Features & Use Cases - Canonical Parity Queries: Provides deduplicated BigQuery SQL templates using the 3-column _kfkevh key (_meta_namespace, _meta_partition_id, _meta_sequence_number) with mandatory partition pre-filtering and payload timestamp alignment. - Unity Catalog Mapping Matrix: Maps 61 canonical BigQuery tables to their Databricks counterparts, including the segregated Shiba tenant (bliplayer_shiba catalog). - CLI Tooling: Runs direct SQL against Databricks via OAuth2 U2M authentication and generates multi-table 3-hour comparative parity dashboards. - Use Case: Audit whether imt_notifications in BigQuery matches bliplayer.raw.notifications in Databricks for a given hour and EventHub namespace, producing an exact row-count comparison. ## Quick Start Ask the assistant to compare row counts between BigQuery imt_notifications and Databricks bliplayer.raw.notifications for a specific UTC hour and namespace.

Frequently Asked Questions about databricks-bq-parity-checker

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compare row counts between BigQuery and Databricks tables?▼

Filter BigQuery by _meta_enqueued_time with a ±4 hour partition envelope, then apply the exact payload StorageDate window, and deduplicate on _meta_namespace, _meta_partition_id, and _meta_sequence_number. In Databricks, filter by StorageDateDayBR partition and the same StorageDate window, then compare hourly counts.

Why do BigQuery and Databricks row counts mismatch at hour boundaries?▼

BigQuery partitions by broker enqueue time (_meta_enqueued_time) while Databricks filters by the payload StorageDate field. Broker ingestion lag creates artificial drift at window edges, so both sides must be aligned on the payload StorageDate timestamp.

How do I run SQL queries against Databricks from the CLI without a PAT?▼

Use the Databricks CLI with OAuth2 User-to-Machine authentication, since personal access tokens are disabled by organization policy. The DEFAULT profile in ~/.databrickscfg handles automatic token refresh, and scripts/dbx_sql.py executes queries against the serverless SQL warehouse.

How does deduplication work for _kfkevh BigQuery tables?▼

Deduplication uses a 3-column composite key: _meta_namespace, _meta_partition_id, and _meta_sequence_number. Apply ROW_NUMBER() partitioned by these columns ordered by _meta_enqueued_time descending and keep the first row, since the legacy messageKey field is removed in _kfkevh.

Where are Shiba tenant tables stored in BigQuery and Databricks?▼

Shiba tables reside exclusively in the segregated GCP project blip-dpl-prd-sam-i-plt-shs-0 as imt_*_shiba tables. In Databricks they live in the bliplayer_shiba catalog under raw and shibablipraw schemas, never in the standard bliplayer catalog.

Related Skills

Explore More Data & Analytics Skills→
mkgs-databricks-demosmkgs-databricks-demos

databricks-iceberg

Manage and access Apache Iceberg tables within Databricks.

Official
Advanced
Arvo-AIArvo-AI
387

grafana

Ingest Grafana alerts via webhook to trigger Aurora investigation workflows.

Official
Intermediate
ivanshamaevivanshamaev
14

apache-kafka

Configure Apache Kafka topics, producers, consumers, connectors, and monitoring.

Community
Advanced
s1366560s1366560
5

data-engineer

Design and implement scalable data pipelines with Apache Spark, dbt, and Airflow.

Community
Advanced