databricks-zerobus-ingest

Build Zerobus Ingest clients that stream records into Databricks Delta tables via gRPC.

Updated Mar 24, 2026
One-click install
npx skills add https://github.com/AarushiShah/coding-agents-databricks-apps --skill databricks-zerobus-ingest-aarushishah
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: databricks-zerobus-ingest
Source: https://github.com/AarushiShah/coding-agents-databricks-apps/tree/main/.claude/skills/databricks-zerobus-ingest
Command: npx skills add https://github.com/AarushiShah/coding-agents-databricks-apps --skill databricks-zerobus-ingest-aarushishah

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-zerobus-ingest-sdk, databricks-sdk, grpcio-tools.

What problem does it solve? Writing near real-time data into Databricks Delta tables normally requires message bus infrastructure like Kafka or Kinesis. This Skill guides you through building Zerobus Ingest producers that write records directly to Unity Catalog managed tables over gRPC, with correct authentication, schema handling, and retry logic. ## Core Features & Use Cases - Multi-language SDK guidance: Working examples for Python, Java, Go, TypeScript, and Rust covering stream creation, record ingestion, ACK handling, and flush/close patterns. - Protobuf schema generation: Generate .proto files from Unity Catalog table definitions, compile language bindings, and apply Delta-to-Protobuf type mappings. - Production operations: Retry with exponential backoff, stream reconnection, at-least-once delivery handling, and throughput limits (100 MB/s, 15,000 rows/s per stream). - Use Case: You have IoT sensor readings that must land in a Delta table within seconds. Use this Skill to create a service principal, generate a Protobuf schema from the table, and deploy a Python producer that ingests records with durability acknowledgments. ## Quick Start Ask the agent to set up a Zerobus Ingest producer in Python that writes records to your Unity Catalog table, starting with endpoint configuration and service principal authentication.

Frequently Asked Questions about databricks-zerobus-ingest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ingest data into Databricks Delta tables without Kafka?

Use the Zerobus Ingest SDK to write records directly to a managed Delta table over gRPC. Create a stream with a service principal, call ingest_record for each record, and wait for durability acknowledgments confirming the write.

How to generate a Protobuf schema from a Unity Catalog table?

Run python -m zerobus.tools.generate_proto with your workspace endpoint, service principal credentials, table name, and output path. The tool produces a .proto file matching the table schema, which you compile with grpcio-tools or protoc.

Should I use JSON or Protobuf with Zerobus Ingest?

Use Protobuf for production workloads because it provides compile-time type checking, forward-compatible schema evolution, and smaller binary payloads. JSON is acceptable for quick prototypes with trivial schemas.

What languages does the Zerobus Ingest SDK support?

The SDK supports Python 3.9+, Java 8+, Go 1.21+, TypeScript/Node.js 16+, and Rust 1.70+. All languages follow the same pattern: initialize the SDK, create a stream, ingest records, handle ACKs, flush, and close.

Why does my Zerobus stream close unexpectedly?

Streams close due to server maintenance, network issues, or zone failures since Zerobus runs in a single availability zone. Implement retry with exponential backoff capped around 30 seconds and reinitialize the stream on connection errors.

What are the throughput limits of Zerobus Ingest?

Each stream supports up to 100 MB/s and 15,000 rows per second, with a 10 MB maximum message size and 2,000 columns per schema. For higher throughput, open multiple streams to the same table from different clients.