databricks-zerobus-ingest

Direct data ingestion into Databricks Delta tables without an intermediate message bus is achieved by using gRPC streaming to write records straight to Unity Catalog managed tables. This approach eliminates operational overhead while maintaining low-latency pipeline execution.

Zerobus ingestion supports building production data producers in Python, Java, Go, TypeScript, and Rust. Each language SDK provides consistent API patterns for streaming records directly into Unity Catalog managed Delta tables.

Generating Protobuf schemas from Unity Catalog tables is handled by the ingestion skill's core logic, which maps Delta table structures to Protobuf definitions. This enables gRPC-based streaming directly from producer applications into your Databricks lakehouse.

Direct Zerobus ingestion provides at-least-once delivery guarantees for real-time IoT data streaming. It includes built-in durability ACKs and retry logic with exponential backoff to automatically recover from transient network outages.

Migrating legacy system data to Databricks Delta tables without intermediate message bus infrastructure is fully supported. You can build producers to stream records directly to Unity Catalog managed tables via gRPC, bypassing the need for Kafka or Kinesis.