bigquery-bigtable-compact-reverse-etl

Use the EXPORT DATA statement with format='CLOUD_BIGTABLE', a URI pointing to your Bigtable table and app profile, and a SELECT that produces a rowkey column plus payload columns. Enable auto_create_column_families or pre-create the column families before running the export.

Serialize rows into a single cell using schemaless Avro binary encoding instead of JSON, which removes repeated field names, then apply Deflate compression. This combination reduces Bigtable SSD storage by roughly 39-50% compared to plain JSON cells.

The project must be attached to a BigQuery reservation with Enterprise or Enterprise Plus edition (or dedicated slots) in the same region as the data. The Bigtable cluster must also reside in the same region or within the dataset's multi-region.

Plain Avro binary saves 10-15% storage at about 5,000 rows per second export throughput, while Avro plus Deflate saves 39-50% but drops throughput to about 2,400 rows per second and doubles slot compute per row. Choose compressed Avro for best cost-benefit on storage-heavy workloads.

Read the cell value, decompress it with zlib.decompress using the -15 raw deflate window, then decode the bytes with fastavro.schemaless_reader against your parsed Avro schema. This yields the original record fields with sub-millisecond latency.

Persistent UDFs stored in a shared utilities dataset are created once and reused by every export query and pipeline, avoiding repeated UDF definitions in each job. This keeps export queries shorter and standardizes the serialization logic across teams.