databricks-zerobus-ingest

You can ingest data into Databricks Delta tables without a message bus by using the Zerobus gRPC API. This approach writes records directly to Delta tables, reducing latency and eliminating infrastructure complexity.

The Zerobus gRPC API supports JSON and Protobuf serialization for Databricks data ingestion. Schema evolution is managed by regenerating Protobuf schemas and recompiling language bindings as needed.

Yes, this direct ingestion method is suitable for high-throughput data from IoT devices, web applications, and batch processing jobs. It provides near real-time precision and handles high volumes efficiently.

Yes, Unity Catalog access is required to ingest data into Delta tables with Zerobus. You also need the Zerobus Ingest SDK and databricks-sdk dependencies to establish the client connection.

Real-time data ingestion handles connection issues and server maintenance through built-in retry and reconnection patterns. It ensures at-least-once delivery and provides durability acknowledgments for ingested records.

The best way to manage schema evolution during Databricks data ingestion is by regenerating Protobuf schemas and recompiling language bindings. This ensures your Delta tables adapt to changing data structures.