databricks-zerobus-ingest

You can ingest data directly into Delta tables via gRPC by using a serverless API to write records into Unity Catalog-managed tables, eliminating the need for traditional message buses. It supports JSON and Protobuf serialization with durability acknowledgments.

Streaming records into Unity Catalog Delta tables without a message bus is achieved through a serverless gRPC ingestion path. This approach provides low-latency data writes and cross-language SDK support for Python, Java, Go, TypeScript, and Rust.

Yes, Unity Catalog ingestion requires a pre-created managed Delta table. You also need a service principal with MODIFY and SELECT grants on that table, and the Zerobus SDK installed in your client environment to establish the gRPC stream.

Yes, you can use Protobuf serialization for near real-time analytics in Delta Lake. The ingestion API supports both JSON and Protobuf formats, applying schema validation and providing durability acknowledgments for reliable streaming workflows.

Schema generation from UC tables for gRPC ingestion allows you to automatically derive the data structure required by the serverless API from your existing Unity Catalog tables. This enables schema validation and ensures incoming records match the Delta table format.

Direct-to-Delta gRPC ingestion requires a pre-existing managed table and a configured service principal with specific grants. While it offers low-latency writes, you must handle retry and reconnection logic on the client side to ensure end-to-end durability during network disruptions.