What problem does it solve?
Build and operate producers that write records directly into Databricks managed Delta tables without requiring a separate message bus, enabling near real-time ingestion with durability acknowledgments and schema validation. The Skill centralizes guidance for authentication, table preparation, SDK usage, Protobuf schema generation from Unity Catalog, and production-safe retry and ACK handling so teams can reliably deliver data into the lakehouse.
Core Features & Use Cases
- Multi-language SDK patterns for Python, Java, Go, TypeScript/Node.js and Rust to create streams, ingest records, wait for ACKs, flush, and close.
- Protobuf generation and compilation from Unity Catalog table definitions to ensure type safety and forward-compatible schemas.
- Operational best practices including service principal grants, environment variable configuration, firewall allowlisting, retry with exponential backoff, and at-least-once delivery handling.
- Use Case: A telemetry pipeline that compiles a Protobuf schema from a Unity Catalog table, spins up multiple producers across regions, and ingests high-throughput sensor data with offset-based durability tracking.
Quick Start
Create a producer that initializes the Zerobus SDK, creates a stream to your Unity Catalog managed Delta table, ingest sample records using JSON or Protobuf, wait for offsets to be acknowledged, then flush and close the stream.
