An agent reading yesterday's warehouse is wrong about today. Confluent and Databricks race to serve the now
Confluent's Streamhouse and Lightning Tables, and a Kafka-free CDC path from Debezium into Databricks, both aim to give agents current business state. Freshness belongs in the agent's specification.
Listen to this article · 4 min
AI-generated narration of the full article.
A customer calls to say a shipment never arrived. The service agent checks the order system, sees “in transit”, and reassures them. The carrier marked it lost two hours ago, but that update reaches the warehouse the agent reads tonight.
Nothing in that exchange is a model error. The agent reasoned correctly from a stale view of the business. As agents move from answering questions to taking actions, that kind of mistake becomes the expensive one, and two announcements in early November target it from opposite directions.
Confluent puts the stream at the center
On November 4, Confluent, now an IBM company, introduced Streamhouse, an architecture meant to give agents and applications a governed, continuously updated view of current business state. Its data streaming platform is now organized in five pillars: stream, connect, process, govern and query.
The new piece is in the query pillar. Lightning Tables, generally available on AWS, materializes Kafka data into tables that can be queried with SQL or an API. Confluent says data is servable about 100 milliseconds after it is created, with query latency of 10 to 100 milliseconds and more than 1,000 queries per second. Those figures come from Confluent’s internal testing and vary with configuration. Lightning Tables is read-only, and Confluent positions Streamhouse alongside existing warehouses and lakehouses, not as a replacement. A Real-Time Context Engine serves that context to agents over MCP.
Accenture, quoted in the announcement, says moving from batch and point-to-point integrations to this design cut its time to insight from months to days.
Databricks removes the stream from the middle
A day later, Databricks described the opposite path. The Debezium Server 3.7 release includes a sink for Zerobus Ingest that pushes change data capture events from databases such as PostgreSQL, MySQL, SQL Server, Oracle and MongoDB straight into Unity Catalog tables, with no Kafka cluster in between. NVIDIA contributed an envelope mode and delivery safeguards: the sink waits until Zerobus confirms events are durable before Debezium moves its position in the source log.
The fine print matters for agents. Delivery is at least once, not exactly once. Each event carries an idempotency key, but duplicates must be removed downstream.
How current is the state your agent reads?
YesterdayNow
- Nightly batchTypical warehouse load
- Hourly or micro-batchCommon lakehouse pipelines
- Push CDCChange events land as they commit
- Materialized streamAbout 100 ms, by Confluent's tests
Freshness is a requirement, not a feature
We have argued that an agent’s context needs a specification, and freshness is one of its four properties. These releases make it possible to meet a tight freshness target. They do not tell you what the target should be.
Write it per action:
- Answering questions about policies or history can tolerate a daily load.
- Telling a customer the status of an order or a claim needs minutes at most.
- Acting on stock, credit, fraud or a payment needs the current state, and a check against the system of record right before the action.
Two practical cautions follow. First, at-least-once delivery means an agent can see the same change twice. Any action triggered by an event, such as a refund or a notification, needs its own idempotency key, or it may happen twice. Second, a fast copy is still a copy. For irreversible actions, the agent should read from or confirm with the system that owns the record, as we argued about where authority to commit should live.
Confluent and Databricks will compete on latency figures. The decision that matters is internal: for each thing your agents do, how old is too old?