Success-Plan Services
Bert Farabaugh
If you’re like me, you’ve spent many late nights debating the best way to move data around a distributed system. Whether you're building a fleet of autonomous vehicles or a patient monitoring system, the "plumbing" you choose matters – a lot. It impacts everything from development costs, time to market, scalability, performance, to whether your system fails when it matters most. With today’s emphasis on edge-to-cloud computing, choosing the right infrastructure is half the battle.
So, let's talk about two popular solutions that are used today in distributed computing: DDS® (Data Distribution Service) and Apache Kafka®. While they both handle large scale data, they were developed for very different domains. With AI enablement increasingly driving infrastructure decisions, which is the right choice for deployment environments where time is money and scalability is king?
Whether you're new to Connext or DDS in general, evaluating Kafka versus DDS for an edge application or edge-to-cloud scenario can be a revelatory experience. DDS can provide a startling counterpart to Kafka at the edge, as Kafka wasn’t designed to support the real-time, low latency data needs of edge systems. This article may even help you build a case for choosing a data-centric approach when system scalability needs to be priority one.
Kafka: Built for the Data Center
Apache Kafka is a distributed, high-throughput, and fault-tolerant event broker, designed for building real-time data pipelines and streaming applications. And certainly, Kafka is robust in terms of data ingestion and durability. However, while Kafka is being used in many enterprise data streaming applications today, it has several issues that prevent it from evolving applications into a platform for physical AI and edge systems.
- Due to its broker / server based architecture, it will never be able to provide real-time data sharing at the edge. Latency increases will eliminate it as a solution for these applications that require the right data at the right time.
- It lacks the ability to control and monitor real-time behavior at the granularity required by control systems that operate at the speed of physical processes rather than human processes.
- Its event-centric paradigm is best suited to static, centralized systems, rather than dynamic systems, such as those that incorporate edge and mobile assets.
DDS: Built for Physical AI Systems
The DDS standard (Data Distribution Service for Real-Time Systems) uses a completely decentralized Peer-to-Peer (P2P) model for direct, low-latency data exchange. Ideal for complex distributed systems and real-time applications, benchmarks indicate that Connext can achieve local message latency as low as 39 microseconds, compared to Kafka's 2,047 microseconds. For applications where latency spikes can be detrimental, the DDS architecture provides a more reliable solution for the rigorous requirements inherent in rapidly evolving edge-to-cloud environments, as well as supporting AI-driven development.
Quality of Service (QoS) is another area where DDS distinguishes itself. DDS provides rich and granular QoS policies, offering over 20 different options to control reliability, durability, deadlines, liveliness, resource limits, and more. This fine-grained control allows developers to tailor the system's behavior to meet specific application requirements.
Kafka, while offering strong durability and replication, has limited QoS options. It supports "at-least-once" delivery, with "exactly-once" achievable through careful application design. For applications where reliability over unreliable networks is critical, DDS’s configurable reliability protocols provide a more robust solution.
DDS is also data-centric rather than event-centric or message-centric. With data-centricity, distributed applications can share state seamlessly from the edge to the cloud, even in highly dynamic environments. Applications and AIs can discover what data is available to make the best decisions based on a live system’s composition.
One additional advantage of DDS is its ability to provide a very scalable and filterable connection from edge systems back up to the cloud. Data needs the ability to be filtered, as it's impossible to send all edge data up to the cloud. This capability makes DDS uniquely suited to supporting the data needs of many critical applications both today and in the future.
Click image to enlarge
Operational Costs and Development Complexity
Operational costs and development complexity are important considerations when choosing between DDS and Kafka. DDS, being an open standard, ensures consistency, portability, and interoperability between different vendor implementations. DDS abstracts away low-level networking details, simplifying the development process. In addition, since DDS handles all your edge data as well as the ability to stream selected data up to the cloud, then the use of a single data model and single infrastructure can be leveraged. This also allows for the elimination of any costly gateways that just end up adding more complexity and potential break points.
Using Kafka in a highly distributed system comes with its own set of challenges. It requires significant expertise in distributed systems concepts and operational complexity for cluster management. Additionally, while Kafka supports TLS for transport-level protection, it lacks end-to-end encryption, with brokers able to see all messages. These factors can introduce hidden costs related to security auditing and maintenance.
Scenarios and Use Cases: When to Choose DDS or Kafka
The decision to choose DDS or Kafka hinges on the primary function and environment of the system. DDS is the best fit for building real-time control systems where deterministic and ultra-low latency are critical. Applications such as autonomous driving, patient monitoring, and defense systems benefit from DDS’s robust, reliable, and low-latency communication.
Kafka, on the other hand, excels in scenarios where high throughput and durability are more important than predictable low latency. It is ideal for building data pipelines, streaming analytics, or replacing traditional message queues for microservices decoupling. When managing long-term, append-only event logs, Kafka’s centralized broker model and strong data persistence capabilities provide significant advantages.
Ultimately, the choice between DDS and Kafka should be guided by the specific requirements of your application. Understanding the key differences in system architecture, real-time performance, data management, QoS, and operational costs will enable you to make an informed decision that best meets your system architecture needs.
Ready to find out how RTI software can help you create your ideal system architecture? Explore our free trial of Connext here.
About the author:
Bert Farabaugh is the Director of Field Application Engineering at RTI. During his 23-year tenure with RTI, Bert has developed extensive expertise in real-time applications and distributed systems architecture design. Prior to his current role, he also spent 12 years developing robotic systems with various key defense contractors. Bert holds a B.S. in Electrical Engineering from The University of Pittsburgh.