databricks-zerobus-ingest

You can ingest near real-time records into Databricks Delta tables by using Zerobus over gRPC streams. This approach pushes records directly into Unity Catalog managed Delta tables while providing durability acknowledgments and production-ready client patterns without requiring a standalone message bus.

You can generate and compile Protobuf schemas from existing Unity Catalog table definitions using the Zerobus ingest workflow. This Protobuf-first approach ensures type-safe, high-performance record ingestion by enforcing strict matching between your record schema and the table schema.

Zerobus gRPC streaming supports building producers and generators across Python, Java, Go, TypeScript, and Rust. This multi-language compatibility allows developers to implement resilient streaming ingestion clients in their preferred development environment.

Pushing records to Unity Catalog managed Delta tables via Zerobus requires OAuth service principal authentication. You must configure your Zerobus server endpoint and service principal credentials before running a producer to ingest Protobuf records into your managed Delta table.

To handle ACK and retry logic for Databricks streaming ingestion, configure your Zerobus stream with built-in operational guardrails. These guardrails manage ACK handling, retries, reconnection, and at-least-once delivery semantics to make your producers resilient during record ingestion.

Yes, you must pre-create Delta tables before starting Zerobus data ingestion. Zerobus does not handle table creation, so you need to set up your Unity Catalog managed Delta tables in Databricks beforehand to ensure strict schema matching during record streaming.