Leading Applications and Architecture for the Industrial IoT (Pt.3)

In Episode 8 of The Connext Podcast:

• [0:00] IoT Protocols
• [0:33] MQTT
• [2:25] XMPP
• [4:05] DDS
• [6:58] AMQP
• [8:17] OPC UA
• [9:30] The Bottom Line: How Do You Choose?
• [14:33] Section 4: Connectivity Architecture for the IIoT
• [16:33] The N-Squared Challenge
• [19:48] The IIRA Core Connectivity Standard
• [23:05] The IIoT Approach to Security
• [24:07] DDS as a Core Standard
• [25:20] DDS Unique Features
• [27:30] The DDS Data Bus
• [30:29] Section 5: Data-Centricity Makes DDS Different
• [34:08] Section 6: The Future of the IIoT

Related Content:

• [eBook] Leading Applications and Architecture for the Industrial Internet of Things, By Stan Schneider
• [Podcast] EP 06 - Leading Applications and Architecture for the Industrial IoT (Pt. 1)
• [Podcast] EP 07 - Leading Applications and Architecture for the Industrial IoT (Pt. 2)

Podcast Transcription:

[0:00] IoT Protocols

Lacey Trebaol: Hi, everyone, and welcome to episode eight of The Connext Podcast. I'm Lacey Trebaol, and today, I'm really excited to bring to you part three of our three-part series where we're providing you all with an audio version of our very popular eBook Leading Applications and Architecture for the Industrial Internet of Things. The author of this eBook is our CEO Stan Schneider. We hope you enjoy.

[0:33] MQTT

MQTT targets device data collection, shown in figure 22. As the name states, the main purpose is telemetry or remote monitoring. Its goal is to collect data from many devices and transport that data to the IT infrastructure. It targets large networks of small devices that need to be monitored or controlled from the cloud. MQTT makes little attempt to enable device-to-device transfer, nor to fan out the data to many recipients. Since it has a clear, compelling single application, MQTT is simple, offering few control options. It also doesn't need to be particularly fast. In this context, real-time is typically measured in seconds.

A hub-and-spoke architecture is natural for MQTT. All the devices connect to a data concentrator server. Most applications don't want to lose data regardless of how long retries take, so the protocol works on top of TCP, which provides a simple reliable stream. Since the IT infrastructure uses the data, the entire system is designed to easily transport data into enterprise technologies like ActiveMQ and ESBs.

MQTT targets applications like monitoring an oil pipeline for leaks or vandalism. Those thousands of sensors must be concentrated into a single location for analysis. When the system finds a problem, it can take action to correct that problem. Other applications for MQTT include power usage monitoring, lighting control, and even intelligent gardening. They share a need for collecting data for many sources and making it available to the IT infrastructure.

[2:25] XMPP

XMPP was originally called Jabber. It was developed for instant messaging to connect people to other people via text messages. XMPP stands for Extensible Messaging and Presence Protocol. Again, the name belies the targeted use. Presence, meaning people, are intimately involved. XMPP uses the XML text format as its native type making person-to-person communications natural. Like MQTT, it runs over TCP, or perhaps HTTP over on top of TCP. Its key strength is its name@domain.com addressing scheme. That helps connect the needle in the huge Internet haystack.

In the IoT context, XMPP offers easy ways to address a device. This is especially handy if the data is going between distant, mostly unrelated points just like the person-to-person case. It's not designed to be fast. In fact, most implementations use polling or checking for updates only on-demand.

A protocol called bidirectional-streams over synchronous HTTP or BOSH lets servers push messages, but real-time to XMPP is on a human scale measured in seconds. XMPP provides a great way, for instance, to connect your home thermostat to a web server so you can access it from your phone. Its strengths in addressing security and scalability make it appropriate for consumer IoT applications.

[4:05] DDS

In contrast to MQTT and XMPP, DDS targets devices that directly use device data. It distributes data to other devices. DDS's main purpose is to connect devices to other devices. It is a data-centric middleware standard with roots in high performance, defense, industrial, and embedded applications. DDS can efficiently deliver millions of messages per second to many simultaneous receivers. Devices demand a data very differently than the IT infrastructure demands it. First, devices are fast. Real-time may be measured in milliseconds or microseconds. Devices need to communicate with many other devices in complex ways, so TCP's simple and reliable point-to-point streams are far too restrictive; instead, DDS offers detailed quality-of-service control, multicast, configurable reliability, and pervasive redundancy.

In addition, fan-out is a key strength. DDS offers powerful ways to filter and select exactly which data goes where, and where can be thousands of simultaneous destinations. Some devices are small, so there are lightweight versions of DDS that run in constrained environments.

Hub and spoke is completely inappropriate for device data use; rather, DDS implements direct device-to-device bus communication with the relational data model. This is often termed a data bus because it is the networking analog to a database. Similar to the way a database controls access to stored data, a data bus controls data access and updates many simultaneous users. This is exactly what many high-performance devices need to work together as a single system.

High-performance integrated device systems use DDS. It is the only technology that delivers the flexibility, reliability, and speed necessary to build complex, real-time applications. DDS is very broadly used. Applications include wind farms, hospital integration, medical imaging, autonomous planes and cars, rail asset tracking, automotive test, smart cities, communications, data center switches, video sharing, consumer electronics, oil and gas drilling, ships, avionics, broadcast television, air traffic control, SCADA, robotics, and defense. RTI has experience with nearly 1,000 applications. DDS connects devices together into working distributed applications at physics speeds.

[6:58] AMQP

AMQP is sometimes considered an IoT protocol. AMQP is all about queues. It sends transactional messages between servers. As a message-centric middleware that arose from the banking industry, it can process thousands of reliable queued transactions. AMQP is focused on not losing messages. Communications from the publishers to exchanges and from queues to subscribers use TCP, which provides strictly reliable point-to-point connection. Further, end points must acknowledge acceptance of each message. The standard also describes an optional transaction mode with a formal multiphase commit sequence.

True to its roots in the banking industry, AMQP middleware focuses on tracking all messages and ensuring each is delivered as intended, regardless of failures or reboots. AMQP is mostly used in business messaging. It usually defines devices as mobile handsets communicating with back office data centers. In the IoT context, AMQP is most appropriate for the control plane or server-based analysis functions.

[8:17] OPC UA

OPC UA is an upgrade of the venerable OPC (OLE for Process Control) protocol. OPC is operational in thousands of factories all over the world. Traditionally, OPC was used to configure and query plant-floor servers. Usually, programmable logic controllers are PLCs. Actual device-to-device communications were then affected via a hardware-based field bus such as Modbus or PROFINET. OPC UA retains some of that flavor. It connects and configures plant-floor servers. The UA version adds better modeling capabilities; thus, a remote client, such as a graphical interface, can browse the device data controlled by a server on the floor. By allowing this introspection across many servers, clients can build a model of the address space of all the devices on the floor.

OPC UA specifically targets manufacturing. It connects applications at the shop-floor level as well as between the shop floor in the enterprise IT cloud. In the taxonomy previously mentioned, it targets systems that require runtime integration.

[9:30] The Bottom Line: How Do You Choose?

The IoT needs many protocols. Those outlined here differ markedly. Perhaps, it's easiest to categorize them along a key few dimensions: quality of service, addressing, and application.

QoS control is a much better metric than the overloaded real-time term. QoS control refers to the flexibility of data delivery. A system with complex QoS control may be harder to understand and program, but it can build much more demanding applications. For example, consider the reliability quality of service. Most protocols run on top of TCP, which delivers strict, simple reliability. Every bite put into the pipe must be delivered to the other end, even if it takes many retries. This is simple and handles many common tasks, but it doesn't allow timing control. TCP's single lane traffic backs up if there's a slower consumer.

Because it targets device-to-device communications, DDS differs markedly from the other protocols in QoS control. In addition to reliability, DDS offers QoS control of liveliness, when you discover problems, resource usage, discovery, and even timing.

Next, discovery. Finding the data needle in the huge IoT haystack is a fundamental challenge. XMPP shines for single-item discovery. Its user@domain addressing leverages the Internet's well-established conventions; however, XMPP doesn't easily handle large datasets connected to one server. With its collection to a server design, MQTT handles that case well. If you can connect to the server, you're on the network. AMQPQs act similarly to servers but for S-to-S systems. Again, DDS is an outlier. Instead of a server, it uses a background discovery protocol that automatically finds data. DDS systems are typically more contained. Discovery across the wide-area network (WAN) or huge device sets requires special consideration.

OPC UA specializes in communicating the information about the system, its configuration, typology, and data context, (the metadata). These are exposed in the collective address base of the individual OPC UA servers. This data can be addressed by OPC UA clients. They can see what is available and choose what to access. OPC UA is not designed for flexible device-to-device interaction.

Perhaps the most critical distinction comes down to the intended applications. Inter-device data use is a fundamentally different use case from device data collection. For example, turning on your light switch remotely, best for XMPP is worlds apart from generating that power, DDS, monitoring the transmission lines, MQTT, or analyzing the power usage back at the data center, AMQP. Of course, there is still confusion. For instance, DDS can serve and receive data from the cloud, and MQTT can send information back out to devices; nonetheless, the fundamental goals of all five protocols differ, the architectures differ, and the capabilities differ. All of these protocols are critical to the rapid evolution of the IoT and all have a place. The IoT is a big place with room for many protocols.

To make confusion even worse, many applications integrate many subsystems, each with different characteristics. With this variety, what is the best path? Experience suggest that designers first identify the applications toughest challenge, and then choose the technology that best meets that single challenge. This is a critical decision. Choose carefully and without prejudice of what you know. The aforementioned requirements-based dimensional decomposition can help. After this step, the choice is usually fairly obvious. Most applications contain a key challenge that clearly fits better with one or the other.

Once the hardest challenge is met, the best way to cover the rest of the application is usually to push your initial choice as far as it will go. There are many bridging technologies, so it is possible to mix technologies; however, it's usually easier to avoid multi-protocol integration steps when possible.

In the long term, the technologies will offer better interoperability. The IIC's connectivity core standard design described later is a key approach. Regardless of your initial choice, the vendor communities are working to provide a non-proprietary path to interoperability.

[14:33] Section 4: Connectivity Architecture for the IIoT

There is no way to build large distributed systems without connectivity. Enterprise and human-centric communications are too slow or too sparse to put together large networks of fast devices. These new types of intelligent machines need a new technology. That technology has to find the right data and then get that data where it needs to go on time. It has to be reliable, flexible, fast, and secure. Perhaps not as obviously, it must also work across many types of industries. Only then can it enable the efficiencies of common machine-based and cloud-based infrastructure for the IIoT.

Connectivity faces two key challenges in the IIoT: interoperability and security. Interoperability is a challenge because the IIoT must integrate many subsystems with different designs, vendor equipment, or legacy infrastructures. Security is a challenge because most enterprise security approaches target hub and spoke designs with a natural center of trust. Those designs cannot handle vast networks of devices that must somehow trust each other. The IIC's IIRA addresses both. Ultimately, the IIoT is about building distributed systems, connecting all of the parts intelligently so that the system can perform, scale, evolve, and functional optimally is the crux of the IIRA.

The IIoT must integrate many standards and connectivity technologies. The IIC architecture explicitly blends the various connectivity technologies into an interconnected future that can enable the sweeping vision of a hugely connected new world.

[16:33] The N-Squared Challenge

When you connect many different systems, the fundamental problem is the N-squared interconnect issue. Connecting two systems requires matching many aspects, including protocol, data model, communication pattern, and QoS parameters like reliability, data rate, or timing deadlines. While connecting two systems is a challenge, it is solvable with a special purpose bridge, but that approach doesn't scale. Connecting N systems together requires N-squared bridges. As N gets large, this becomes daunting.

One way to ease this problem is to keep N small. You can do that by dictating all standards and technologies across all systems that interoperate. Many industry-specific standards bodies successfully take this path. For instance, the European Generic Vehicle Architecture, GVA, specifies every aspect of how to build military ground vehicles, from low-level connectors to top-level data models. The German Industrie 4.0 effort takes a similar pass at the manufacturing industry, making choices for ordering and delivery, factoring design, technology, and product planning. Only one standard per task is allowed.

This approach eases interoperation. Unfortunately, the result is limited in scope because the rigidly-chosen standards cannot provide all functions and features. There are simply too many special requirements to effectively cross industries this way. Dictating standards also doesn't address the legacy integration problem. These two restrictions, scope and legacy limits, make this approach unsuited to building a wide-ranging, cross-industry industrial Internet.

On the other end of the spectrum, you can build a very general bridge point. Enterprise web services work this way using an Enterprise Service Bus, ESB, or a mediation bus like Apache Camel; however, despite the bus in its name, an ESB is not a distributed concept. All systems must connect to a single point where each incoming standard is mapped to a common object format. Because everything maps to one format, the ESB requires only one-way translation, avoiding the N-squared problem. Camel, for instance, supports hundreds of adapters that each convert one protocol or data source to and from Camel's internal object format; thus, any protocol can, in principle, connect to any other.

Unfortunately, this doesn't work well for demanding industrial systems. The single ESB service is an obvious choke and failure point. ESBs are large, slow programs. In the enterprise, ESBs connect large-grained systems executing only a few transactions per second. Industrial applications need much faster, reliable, smaller-grained service, so ESBs are not viable for most IIoT uses.

[19:48] The IIRA Core Connectivity Standard

The IIRA takes an intermediate approach. The design introduces the concept of a connectivity core standard. Unlike an ESB, the core standard is very much a distributed concept. Some endpoints can connect directly to the core standard. Other endpoints and subsystems connect through gateways. The core standard then connects them all together. This allows for multiple protocols without having to bridge between all possible pairs. Each needs only one bridge to the core.

Like an ESB, this solves the N-squared problem, but, unlike an ESB, it provides a fast, distributed core, replacing the centralized service model. Legacy and less-capable connectivity technologies transform through a gateway to the core standard. There are only N transformations, where N is the number of connectivity standards.

Obviously, this design requires a very functional core connectivity standard. Some systems may get by with slow or simple cores, but most industrial systems need to identify, describe, find, and communicate a lot of data with demands unseen in other contexts. Many applications need delivery in microseconds or the ability to scale to thousands or even millions of data values and nodes. The consequences of a reliability failure can be severe. Since the core standard really is the core of the system, it has to perform.

The IIRA specifies the key functions that connectivity framework and its core standard should provide: data discovery, exchange patterns, and quality of service. QoS parameters include delivery reliability, ordering, durability, lifespan, and fault tolerance functions. With these capabilities, the core connectivity can implement the reliable, high-speed, secure transport required by demanding applications across industries.

The IIRA outlines several data QoS capabilities for the connectivity core standard. These ensure efficient, reliable, secure operation for critical infrastructure. Data quality of service. Number one, delivery: provide reliability and redelivery. Number two, timeliness: prioritize and inform when information is late. Three, ordering: deliver in the order produced or received. Four, durability: support late joiners, survive failures. Five, lifespan: expires stale information. Six, fault tolerance: enable redundancy and failover. Seven, security: ensure confidentiality, integrity, authenticity, and non-repudiation.

[23:05] The IIoT Approach to Security

Security is also critical. To make security work correctly, it must be intimately married to the architecture. For instance, the core standard may support various patterns and delivery capabilities. The security design must match those exactly. For example, if the connectivity supports publish/subscribe, so must security. If the core supports multicast, so must security. If the core supports dynamic plug-and-play discovery, so must security. Security that is this intimately married to the architecture can be imposed at any time without changing the code. Security becomes just another controlled QoS, albeit more complexly configured. This is a very powerful concept. The integrated security must extend beyond the core. The IIRA allows for that too. All other connectivity technologies can be secured at the gateways.

[24:07] DDS as a Core Standard

The IIRA does not currently specify standards. The IIC will take that step in the next release. However, it's clear that the DDS standard is a great fit to the IIRA for many applications. DDS provides automated discovery, each of the patterns specified in the IIRA, all the QoS settings, and intimately integrated security.

This is no accident. The IIRA connectivity design draws heavily on industry experience with DDS. DDS has thousands of successful applications in power systems: huge hydropower dams, wind farms, and micro grids; medicine: imaging, patient monitoring, emergency medical systems; transportation: air traffic control, vehicle control, and automotive testing; industrial control: SCADA, mining systems, PLC communications; and defense: ships, avionics, autonomous vehicles. The lessons learned in these applications were instrumental in the design of the IIRA.

[25:20] DDS Unique Features

DDS is not like other middleware. It directly addresses real-time systems. It features extensive fine control of real-time QoS parameters including reliability, bandwidth control, delivery deadlines, liveliness status, resource limits, and security. It explicitly manages the communication data models or types used to communicate between endpoints. It is thus a data-centric technology.

Like a database, which provides data-centric storage, DDS understands the contents of the information it manages. DDS is all about the data. This data-centric nature, analogous to a database, justifies the term data bus. At its core, DDS implements a connectionless data model with the ability to communicate data with the desired QoS.

Originally, DDS focused on publish/subscribe communications. Participants were either publishers of data or subscribers to data. Later versions of the specification added request-reply as a standard pattern. Currently, RTI also offers a full queuing service that can implement "one of N" patterns for applications like load balancing.

The key difference between DDS and other approaches is not the publish/subscribe pattern. The key difference is data-centricity.

A DDS-based system has no hard-coded interactions between applications. The data bus automatically discovers and connects publishing and subscribing applications. No configuration changes are required to add a new smart machine to the network. The data bus matches and enforces quality of service.

DDS overcomes problems associated with point-to-point system integration such as lack of scalability, interoperability, and the ability to evolve the architecture. It enables play-and-play simplicity, scalability and exceptionally high performance.

[27:30] The DDS Data Bus

The core architecture is a data bus that ties all the components together with strictly controlled data sharing. The infrastructure implements full QoS control over reliability, multitask, security, and timing. It supports fully-redundant sources, syncs networks and services to ensure highly reliable operation. It needs no communication servers for discovery or configuration; instead, it connects data sources and syncs through a background meta traffic system that supports massive scale with no servers.

The data bus technology scales across millions of data paths, ensures ultra reliable operation, and simplifies application code. It does not require servers, greatly easing configuration and operations while eliminating failure and choke points.

DDS is by far the most proven technology for reliable, high-performance, large-scale IIoT systems. As of this writing, there are at least 12 implementations, several of these backed by commercial vendors. RTI provides the leading implementation.

Conceptually, DDS is simple. Distributed systems must share information. Most middleware works by simply sending that information to others as messages. DDS sends information too, of course; however, DDS conceptually has a local store of data, which looks to the application like a simple database table. When you write, it goes into your local store, and then the messages update the appropriate stores on remote nodes. When you read, you just read locally. The local stores together give the applications the illusion of a global data store.

Importantly, this is only an illusion. There is no global place where all the data lives. That would be a database. In a data bus, each application stores locally only what it needs and only for as long as it needs it; thus, DDS deals with data in motion. The global data store is a virtual concept that in reality is only a collection of transient local stores. DDS also matches producers and consumers to ensure proper operation. This matching includes data flow rates, remote liveliness, filtering, security, and performance. Once matched, the middleware enforces the contract. For instance, if a subscriber needs updates 1,000 times per second, the middleware will ensure a fast producer is available. The standard defines more than 20 of these QoS policies.

[30:29] Section 5: Data-Centricity Makes DDS Different

Systems are all about the data. Distributed systems must also share and manage that data across many processors and applications. The strategy to understand and manage this state is a fundamental design decision. Data centricity can be defined by these three properties.

One: the interface is the data. There are no artificial wrappers or blockers to that interface like messages, objects, files, or access patterns. Two: the infrastructure understands that data. This enables filtering and searching, tools and selectivity. It decouples applications from the data and thereby removes much of the complexity from the applications. Three: the system manages the data and imposes rules on how applications exchange data. This provides a notion of truth. It enables data lifetimes, data model matching, CRUD interfaces, et cetera.

An analogy with the database, the data-centric storage technology is instructive. Before databases, storage systems were files with application-defined ad hoc structure. A database is also a file, but it's a very special file. A database has known structure and access control. A database defines truth for the system. Data in the database can't be corrupted or lost. By enforcing structure and simple rules that control the data model, data bases ensure consistency. By exposing the structure to all users, databases greatly ease system integration. By allowing discovery of data and schema, databases also enable generic tools for monitoring, measuring, and mining information.

Like a database, data-centric middleware imposes known structure on the transmitted data. The data bus also sends messages, but it sends very special messages. It sends only messages specifically needed to maintain state. Clear rules govern access to the data, how data in the system changes, and when participants get updates. Importantly, the infrastructure sends messages to the applications. The systems looks like a controlled global data space. Applications interact directly with the data and data properties like age and rate. There is no application level awareness or concept of message.

With knowledge of the structure and demands on the data, the infrastructure can do things like filter information, selecting when or whether to do the updates. The infrastructure itself can control QoS-like update rate, reliability, and guaranteed notification of peer liveliness.

The infrastructure can discover data flows and offer those to applications and generic tools alike. This knowledge of the data status in a distributed system is a crisp definition of truth. As in databases, this accessible source of truth greatly eases system integration. The structure also enables tools and services that monitor and view information flow, route messages, and manage caching.

[34:08] Section 6: The Future of the IIoT

The IIoT is clearly in its infancy. Like the early days of the Internet, the most important IIoT applications are not yet envisioned. The killer application that drove the first machine-to-machine connections for the Internet was email; however, once connected, the real power of distributed systems created an entirely new ecosystem of value. This included web pages, search, social media, online retail and banking, and much more.

The real power of the Internet was barely hinted at in its early days. The IIoT will likely follow a similar pattern. Today, many companies are most focused on collecting data from industrial systems and delivering it to the cloud for analysis. This is important for predictive maintenance, system optimization, and business intelligence. This killer app is driving the initial efforts to build connected systems; however, the future holds much more promise than optimizing current systems. By combining high-quality connectivity with smart-learning and machine intelligence, the IIoT future holds many new systems that will revolutionize our world. It will, for instance, save hundreds of thousands of lives a year in hospitals, make renewable energy sources truly practical, and completely transform daily transportation.

Projections of the economic and social benefits range greatly but all agree the impact is measured in the multiple trillions of dollars in a short 10 years. That is a daunting but inspiring number. The IIoT will be a daunting but inspiring transformation across the face of industry.

Thanks for listening to episode eight of our eBook, Leading Applications and Architectures for the Industrial Internet of Things. We hope you enjoyed it. If you have any questions, please reach out. You can reach us at podcast@rti.com or over on social media. Thanks for listening to The Connext Podcast. Have a great day.