Zerobus Ingest Explained: Streaming Events Straight into Your Lakehouse Without Running Kafka

Zerobus Ingest is Databricks' answer to a question many data teams quietly ask: why are we running a message bus when the lakehouse is the only place our events ever go? Part of Lakeflow Connect, it lets applications, devices and services push records directly into Unity Catalog Delta tables. Databricks announced a public preview on 30 October 2025 and declared it generally available on 23 February 2026, on AWS and Microsoft Azure.

Below: the zero-bus idea, how the service works, and where you should still keep your bus.

The message bus tax#

A typical pipeline: producers write to Kafka, Kinesis or Event Hubs, a connector or Structured Streaming job reads the topic, and data lands in a Delta table. With several consumers, that design earns its keep.

But in a surprising number of estates the bus has exactly one consumer: the lakehouse. The team still pays for:

• Brokers, partitions, retention settings and cluster upgrades
• A schema registry and connector fleet to keep in step
• Consumer jobs that must run continuously and be monitored
• Specialist Kafka knowledge on call

That is the message bus tax: real operational cost for a component that only stages data. Databricks calls this a "two-hop" architecture.

What "zero-bus" means as a pattern#

The general pattern, useful whatever platform you run, is simple: when the analytical store is the only sink, let producers write to it directly through a durable, managed ingestion endpoint.

For the pattern to be safe, the endpoint must provide what the bus used to:

• Durable acknowledgement, so a producer knows a record will not be lost
• Schema validation at the door, so bad data is rejected early
• Batching and file optimisation, so millions of small writes do not become millions of small files
• Authentication and access control at the table level

Without those, writing straight to the warehouse is a fragile shortcut.

How Databricks Zerobus Ingest works#

You create a Delta table in Unity Catalog, grant a service principal access, and producers write to a single managed endpoint. According to the Databricks documentation, the service validates records against the table schema, makes accepted records durable, acknowledges them, and then materialises them into the table shortly afterwards.

What Databricks documents as of today:

• Interfaces: gRPC through SDKs, REST and OpenTelemetry (OTLP) are listed as generally available; MQTT and Kafka-compatible APIs are in Beta. Check the release-stages page for your cloud, as stages move.
• SDKs: the GA announcement lists Python, Java, Rust, Go and TypeScript; the docs note C++ and C# SDKs in Beta.
• Record formats: JSON, Protocol Buffers (recommended for production) and Apache Arrow via Arrow Flight. The REST API currently accepts JSON only.
• Delivery semantics: at-least-once, provided the interface returns a durability acknowledgement and the client retries anything unacknowledged. For MQTT, only QoS 1 gives that guarantee.
• Targets: managed Delta tables and streaming tables. Ingesting into tables on default storage is in public preview.
• Schema: not auto-evolved. You evolve the table first, then update producers. Non-matching records are rejected unless you use the rescue column (Beta).

On performance, Databricks states up to 100 MB/s per connection, over 10 GB/s aggregate into a single table and thousands of concurrent clients. The quotas page gives roughly 150 ms time-to-durability and about five seconds time-to-table, with several limits adjustable on request. The maximum message (or batch) size is 10 MB. Pricing, per the announcement, is volume-based under the Lakeflow Jobs Serverless SKU; Databricks has not published a rate in the materials we reviewed.

A simplified producer loop looks like this. Illustrative pseudocode, not the exact SDK API. Follow the official SDK reference for real code:

stream = zerobus.open_stream(endpoint, table="main.iot.sensor_readings", credentials=service_principal)
for chunk in readings.batched(1000):
    for r in chunk:
        last_offset = stream.ingest(r)          # queued, returns an offset
    stream.wait_for_offset(last_offset)          # confirm durability once per chunk
stream.close()                                    # flushes remaining records

Confirm durability per chunk, not per record; blocking on every record throws away throughput.

Architecture before and after#

Aspect	Bus-based pipeline	Zerobus Ingest
Hops to the table	Producer → broker → consumer job → Delta	Producer → managed endpoint → Delta
Components you operate	Brokers, partitions, schema registry, connectors, streaming job	Table, service principal, producers
Scaling unit	Partitions	Connections (service autoscales)
Governance	Split across bus ACLs and Unity Catalog	Unity Catalog on every write
Replay from retained log	Yes, within retention	No bus log to replay from
Multiple independent consumers	Native	Not the purpose
Delivery guarantee	Depends on configuration	At-least-once with acknowledgements and retries

If you are new to the lakehouse side of this picture, our Databricks lakehouse guide covers Delta tables and Unity Catalog in more depth.

When to use Zerobus, and when you still need a bus#

Databricks is explicit that this is a single-sink design. Its product page says it does not replace a message bus when multiple operational consumers need the same data, as in microservice communication.

Zerobus is a strong fit when:

• The lakehouse is the only (or overwhelmingly main) destination
• Producers are devices, edge gateways, ephemeral web apps or on-premises systems that can push but cannot buffer or write to object storage
• You want governed tables in seconds, not a near-real-time pipeline you have to babysit

Keep a bus when you need:

• Fan-out: several services, search indexes or caches consuming the same events
• Replay: re-reading a retained log to rebuild a downstream system
• Event-driven microservices: services reacting to each other, not just to analytics
• Exactly-once end-to-end semantics that your existing stack already provides

Databricks also points Kafka users needing a lakehouse copy to managed streaming connectors, files in cloud storage to Auto Loader, and sub-second needs to real-time mode.

Use cases: IoT, clickstream and telemetry#

• IoT and manufacturing. Gateways or forwarding agents push sensor readings straight to a table. Databricks cites Toyota, which processes telemetry from thousands of factory devices, and Joby Aviation, which streams aircraft telemetry. This is a natural match for our IoT engineering work, where edge devices often cannot host a Kafka client sensibly but can make an authenticated gRPC, REST or MQTT call.
• Clickstream. Web and mobile events for personalisation and A/B testing, written to Delta and queryable within seconds.
• Observability and security. With OTLP support, logs, metrics and traces can be sent to the lakehouse as a configuration change, which suits threat detection and long-term telemetry analysis. Once landed, these tables feed the kind of fast analytical workloads described in our OLAP explainer.

Governance with Unity Catalog#

Removing the bus removes a second permission model: every write targets a Unity Catalog table, so lineage and fine-grained access control apply from the first record.

• Producers authenticate as a service principal using OAuth client credentials; REST tokens expire hourly and must be refreshed.
• The principal needs USE CATALOG, USE SCHEMA, and SELECT plus MODIFY on the target table. Grant nothing broader.
• Use one principal per producer class (factory gateways, web tier, OTel collector) so you can revoke and audit cleanly.

Implementation checklist#

1. Confirm the single-sink assumption. List every consumer of the events. If there is more than one operational consumer, stop and reconsider.
2. Check region and release stage for your cloud and each interface you plan to use.
3. Design the table first: Protobuf schema, partitioning or liquid clustering, and a rescue column if you expect drift.
4. Plan schema evolution: alter the table, then roll producers.
5. Make producers idempotent-friendly. At-least-once means retries can create duplicates; include an event ID and deduplicate downstream.
6. Confirm durability per chunk with offsets or flush, and always close streams cleanly.
7. Load test against the quotas and request increases before go-live.
8. Monitor rejections and retries as first-class metrics.

Our data engineering and cloud and DevOps teams usually run steps 1 to 3 as a short assessment first.

Risks to weigh#

• Duplicates. At-least-once is not exactly-once. Budget for deduplication.
• No replay log. If a downstream transformation is wrong, you rebuild from the Delta table and its history, not from a bus.
• Durability scope. A Databricks community post on the GA release noted single availability-zone durability at launch, with multi-AZ on the roadmap. Check the current position for regulated workloads.
• Buffered data retention. The quotas page states unflushed buffered data is deleted after 28 days.
• Platform coupling. Producers now target a Databricks endpoint; a second sink later may mean reintroducing a bus.
• Preview and Beta pieces. MQTT, Kafka-compatible APIs, the rescue column and default-storage tables are not yet GA.

FAQs#

Is Zerobus Ingest generally available?#

Yes. Databricks announced general availability on 23 February 2026 on AWS and Azure. Some interfaces and features, such as MQTT and the Kafka-compatible APIs, remain in Beta or preview.

Does Zerobus Ingest replace Kafka?#

Only where Kafka exists purely to land data in the lakehouse. If you need fan-out, replay or event-driven microservices, keep the bus.

What delivery guarantee does it offer?#

Databricks documents at-least-once delivery when the client waits for durability acknowledgements and retries unacknowledged records. Plan for possible duplicates.

Can IoT devices write to it directly?#

Yes, via the SDKs, REST or MQTT (Beta). For constrained fleets, an edge gateway simplifies credentials and batching.

Planning a lakehouse ingestion redesign? Talk to VCNR Technologies.