The Dawn of Physical Data Streaming

The smart-planet promise is everywhere these days: Robotaxi fleets. Digital ORs. Golden Dome. Smart farms. Dark factories. These integrated systems would connect dozens or thousands of smart components together into a working whole.

Unfortunately, while there are a few custom-built examples, these systems require integration that’s beyond today’s technology. But not for long. Physical data streaming is becoming real.

What is Data Streaming?

Data streaming is the backbone of digital enterprise. Cloud-based streaming platforms from leaders such as Confluent (now IBM), Google, Microsoft, and AWS connect together applications, AIs, humans, and databases. Business networks, the Web, AI platforms, and SaaS applications would simply not exist without these platforms.

If you are an IT (Information Technology) pro, you already know this. But this blog is for those in OT (Operational Technology), the edge computing for industries such as MedTech, defense, automotive, and industrial automation. Streaming technologies for these OT systems, aka Physical Data Streaming, are only now becoming sophisticated enough to qualify as true platforms.

For clarity, we are not talking about how you watch Netflix or listen to Spotify; those applications incrementally download existing content so your local device can play a continuous movie or song. That’s called “streaming,” but it’s a very different concept from cloud data streaming as infrastructure.

Cloud data streaming frees applications to work together in real time. Rather than storing information in a database for later processing, streaming platforms move between applications immediately. Early data-streaming solutions were simply “protocols” that enabled basic interoperability.

Today’s platforms do far more than move data. Cloud streaming systems mostly share and process “logs,” streams of text records that capture events and actions so applications can process them. They are much more than just a protocol; streaming platforms implement 4 major functional layers, often called “planes”:

• Data plane: Handles the movement and storage of streaming data, including named topics, partitions, throughput scaling, ordering, fan-out, durability, retention, and replay.
• Trust plane: Ensures reliability and security through delivery guarantees, schema contracts, access control, tenancy boundaries.
• Operations plane: Provides system visibility and maintenance by providing observability, metrics/logs, debugging, geo recovery, and managed deployment.
• Developer plane: Enables building and integration through APIs, connectors, processing frameworks, governance, automation.

Together, these planes implement a shared-data layer across applications. They enable reliable operation, interacting applications, deep analytics, and increasingly full automation.

However, as powerful as these platforms are, they can’t handle OT challenges. They do many similar things: fundamentally connect data between systems, scale well, and provide reliable infrastructure. They optimize for throughput, durability, and elasticity. But to control physical systems, data sharing has to work in the real world.

That means connecting at physics speeds, handling messy real-world special cases, managing many more data types, dealing with unreliable connections and partial failures, and adapting to hundreds of embedded processors and operating environments. Because of that, cloud-streaming technology’s reach into the physical world extends only to relatively simple “IoT” devices that rarely connect to each other at all. That leaves entire classes of applications beyond reach without prohibitively expensive custom platforms.

What is Physical Data Streaming?

A physical data-streaming platform must connect physical devices together fast enough and flexibly enough to build the smart-world applications on the horizon. Physical data streaming requires most of what the IT systems do. But it also needs much better determinism and timing control, system-wide coherence, system governance, and more. The demands defy listing – the space is so varied and the requirements so broad. Even when there are similar types of requirements, the need can be fundamentally different.

As only one example, cloud systems strive for “five nines” availability, meaning they are “up” 99.999% of the time. But in a year, the remaining 0.001% downtime is still 5 minutes – 300 seconds. Navy ships at battle, surgical robots handling delicate operations, and cars running down the road can’t stop even for a few seconds. Another way to think of it: to avoid a 3-second catastrophic outage, they need to surpass “seven nines” reliability, 100x better than the best of the cloud. That requires a different approach; in a few seconds, you can’t reboot servers, switch networks, or even failover and reconnect backup servers. Physical streaming is hard. Really hard.

Nonetheless, over the last few years, Connext is evolving into a real physical data-streaming platform. It addresses all 4 of the planes:

• Data plane: Connext addresses all the challenges above, plus much finer delivery control, much lower latency, and no single-point-of-failure servers.
• Trust plane: Schema contracts in both cloud and physical systems are different but both complex and always growing. Delivery guarantees are much tighter: real-time in the cloud means “fast enough”; missing timing is expensive and annoying. Real-time at the edge means sub-millisecond, life-or-death failure. Security and access control are harder with so many varied applications. Tenancy management in physical systems is easier, or even not a factor, because there are fewer entities sharing resources.
• Operations plane: Observability and distributed debugging are nascent. Managed deployment is coming…soon.
• Developer plane: Connext sports a powerful routing service with plug-in connectors, transparent language, OS, and chip translation, and standard APIs to access data. With AI support for development, Model Context Protocol (MCP) service for agentic automation, and evolving governance, the developer system is expanding rapidly.

Connext stands apart as a platform purpose-built for physical AI. It provides deterministic, real-time data coordination that cloud-native platforms simply cannot deliver at the edge, where the digital and physical worlds meet.

What’s New? What’s Next?

Our latest release, Connext 7.7, takes several important steps. It improves large-scale discovery of system participants, a key physical system need. It adds remote debugging. It adds redundant and resilient persistence, and it adds native Google Protocol Buffer (GPB) support to ease schema matching. Of course, that’s not all; Connext 7.7 fundamentally offers easier use with Connext AI, simplified development, easier licensing models, included onboarding, and more. Connext 7.7 is just one step in an evolution, of course. The road to the smart-planet future is long.

The evolution will go on. Perhaps most significantly, physical systems are no longer islands on the edge. Physical data must reach beyond the edge and integrate with cloud systems. Many of our customers do this already; we do support cloud connections and can deliver better performance than other approaches. But deep edge-cloud data access, agentic AI integration, much better observability, and easily unified data models will be much easier soon. Stay tuned.

Enterprise data-streaming platforms run the modern online world. We need analogous physical data-streaming platforms to run the modern real world. The smart planet becomes more real every day, but it will stall without reliable, scalable, data infrastructure. I’m excited to see RTI take this path.

About the author:

Stan is CEO of Real-Time Innovations (RTI), the data streaming company for intelligent distributed systems. RTI’s Connext software is the critical nervous system for over 2,000 designs across Aerospace and Defense, MedTech, Automotive, and Robotics. RTI Runs a Smarter World^™.

Stan has a PhD in EE/CS from Stanford with a focus in autonomous systems.