Building out a distributed system infrastructure in today’s emerging Industrial Internet of Things (IIoT) landscape can be a daunting task, to say the least. If you are a developer or system architect, you know that there are many tools and protocols available to use to move data around in your distributed application. Not to mention the possibility of building out your own custom solution directly on TCP or UDP sockets. Wouldn’t it be great if a lot of the work that needed to be done before you could make a decision on your next infrastructure was already done for you?
You know what? The work has been done and is now available to help you make that decision. You must be asking, "Who did all this research and is it biased by some company looking to sell their own solution?" The good news is that the research was completed by an independent consortium, the Industrial Internet Consortium (IIC). It was conducted in a vendor-neutral, unbiased way and the resulting information is now available to you.
Full disclaimer: Yes, I do work for a company that provides an infrastructure for the Industrial Internet, but by no means am I saying that our solution is the best solution. The real answer to the question, "What is the best solution?" is, "It depends."
The answer is dependent on what you need from an infrastructure solution:
- Do you just need to send bytes of data from point A to point B?
- Do you need the data to be delivered reliably?
- Do you need any data-centric awareness so that the infrastructure can provide some efficient delivery decisions for a more intelligent solution?
- How much of the data handling do you want to hand off to the infrastructure?
The answers to these critical questions, and many more, are what I investigate in this post. By the end of this post, hopefully you will have the information you need to make an informed decision on what the best solution is for your unique application.
About the Industrial Internet Consortium (IIC)
The IIC was formed in 2014 by some very big players in the Industrial Internet landscape. The founding companies (Cisco, Intel, AT&T, IBM and GE) set out to create an organization focused solely on the needs of Industrial Internet applications. Now the consortium is made up of over 250 companies that are both big and small. The results from this consortium include a set of documents that outline the needs and potential solutions for these types of Industrial Internet applications. The IIC Industrial Internet Connectivity Framework (IICF) document, a guidance document, is perfect for helping you determine the best solution for market-based examples. In addition to various documents, they have also established testbeds that will be used to prove the ability of various technologies to meet various real-world market examples. Information on the available documents and market-based testbeds can be found on the IIC website.
Delivering Data: Transports and Frameworks
There are many solutions available today for getting data between applications. In the IICF document, these solutions are broken down into two categories: transports and frameworks. Let’s take a look at these two types of data transfer solutions to see where they fit in the overall stack of connectivity layers. Figure 1 below shows this connectivity stack.
Pretty much everyone reading this document has seen a connectivity stack like this, but the IIC's stack has one clear distinction: the transport and framework layers.
Typically, we tend to group all of the solutions you see in the transport and framework categories together, but there is one very big differences between a transport and a framework. A transport is used to deliver data from point A to point B, whereas a framework basically leverages the capability of a transport while providing a data type system for interoperability. Simply put, when using just a transport, the application must formulate the data into a generic buffer for handing off to the transport. However, with a framework, the application just needs to hand off the data to the framework, and the framework will take care of constructing a buffer for the underlying transport to go ahead and send its data. Working at the data level for an application has many benefits for applications providing capabilities like content filtering and discovery, whereas if your application only uses something at the transport layer, it is up to the application to implement discovery and filtering if needed. Table 1 gives you all of the capabilities that are available at each layer of transport or framework.
Can you build a distributed Industrial Internet application using a transport? Yes. Can you build a distributed Industrial Internet application using a framework? Yes. Is one better than the other? The real answer is: "It depends." It depends on the requirements of your application as to which solution is best suited for your infrastructure. The remainder of this post will go through several of these frameworks and transports so that you can decide which is the right technology for your application.
There are many solutions available today for getting data between applications. In the IICF, there are transports called out that leverage the standard IP socket interfaces of either UDP or TCP. If your application needs reliable data transfer, then a developer would pick TCP for its connection-oriented capabilities and reliable mechanisms. For simpler connection and non-reliable data transfer, then UDP would be chosen for its ease of use and multicast delivery. For years, most network applications used these basic interfaces for sending and receiving data. All of the capabilities provided by the higher layer transports (listed in Table 1) would have to be built out directly, within the application. When looking at the higher layer transports of DDS-RTPS, CoAP, MQTT, HTTP and OPC-UA Bin, we will really only look at the details for CoAP and MQTT. The DDS-RTPS, HTTP and OPC-UA Bin transports are basically tied directly to the frameworks above them of DDS, Web Services and OPC-UA respectively. The capabilities of these transports will be discussed as part of the framework discussion to follow.
Let's take a look at Message Queueing Telemetry Transport (MQTT). Again, MQTT is listed here as a transport because it does not enforce nor implement a data model for applications. It only provides a buffer upon which applications must formulate their data for sending and receiving. Its primary purpose for which it was created is listed right in its name: Telemetry. Having a device or application out in the field connect up to and report data to a backend cloud or off-site processing location. This transport is great for things like a home IoT gateway or manager of a set of deployed devices. The primary architecture for MQTT is broker-based as can be seen in Figure 2.
In this architecture, all remote clients send their data to the MQTT broker, and the broker is responsible for sending its data out to any clients that have asked for that data. This broker-based architecture makes it easy for sending and receiving data in a loosely coupled fashion, but it does not lend itself to supporting low latency, highly deterministic industrial applications. As a transport, MQTT has a place in the overall landscape of Distributed Industrial Applications. The following is a tool that you can use to determine if MQTT is something you should use for your next or current project. Here are five "yes" or "no" questions for you. If your answer to three or more of these questions is "yes" then MQTT is the right choice for you.
- Do you think of your application as data collection?
- Is there little device-device communication?
- Is interoperability not a consideration?
- Do you have many small devices?
- Is software a minor challenge?
MQTT is the only transport listed in the IICF document that isn’t really tied to a higher layer framework. This is the reason we broke it out separately as a transport. Now, let’s take a look at the frameworks listed in the IIC document.
As mentioned earlier, the distinguishing difference between a framework and transport is the fact that a framework includes a capability for maintaining and enforcing a data model that is used by the applications participating in the framework. Of the four frameworks called out, OPC-UA, OneM2M, DDS and Web Services, we will only be looking at the first three. Web Services is a very well known framework with plenty of online references that can be found for research, and for that reason, we will not review it in this post. For each of the selected frameworks (OPC-UA, OneM2M and DDS) we will provide an overview of the basics of that framework and also the key questions that must be answered to determine if that framework is the correct solution for your distributed IIoT application.
Let's take a look at OPC-UA as our first framework. The Open Platform Communications Unified Architecture (OPC-UA) is described in the IICF document as, "An industrial communication architecture for platform independent, high performance, secure, reliable, and semantic interoperability between sensors, field devices, controllers, and applications at the shop-floor level in real-time as well as between the shop floor and the enterprise IT cloud."
Like MQTT, OPC-UA is also a broker-based architecture. OPC-UA clients provide data objects to the server and the server then responds to requests from other OPC-UA clients. Typically these data objects are very device-centric objects that are basically a collection of various device input and output variables. A graph of desired objects can be constructed by a client and the server will map out all of its collected objects and present a coherent response to the requesting client. Figure 3 is a typical architecture diagram showing a graph of existing objects available in a given system.
Figure 3. Typical Device Data Objects Graph in OPC-UA
OPC-UA infrastructure is used primarily in Industrial Automation and Manufacturing environments. Here are four questions that should be answered to determine if OPC-UA is the correct solution for your application:
- Are you in discrete manufacturing?
- Are you associated with the German Plattform Industrie 4.0 program?
- Are you building a device that will be integrated by control or process engineers or technicians, rather than software engineers?
- Will your product be used in different applications in different systems, as opposed to one (type of) system where you control the architecture?
- Are you building equipment for a "workcell"?
If you can answer "yes" to three of these questions, then OPC-UA is the right choice for your application.
The second framework called out by the IIC document is oneM2M. The description of one M2M from the IICF document is, "oneM2M provides a common service layer that sits between applications and connectivity transport. It offers functions that IoT applications across different industry segments commonly need. Those functions are exposed to applications via RESTful APIs. OneM2M standards comprise a horizontal platform architecture that fits within a three-layer model comprising of applications, middleware services and networks. OneM2M's connectivity standards permit applications that are hosted on connected machines and devices, enterprise systems and mobile devices to communicate with each other in an efficient, secure manner. The oneM2M horizontal platform is scalable as the Common Service Elements can be deployed on hosts, at the proximal network edge or within the enterprise cloud." The primary applications that use oneM2M today are Home Automation and large-scale applications leveraging mobile systems. The services available in the Common Service Layer are provided by large telecommunications companies. Figure 4 shows the architecture of the three layers called out in oneM2M.
Following are the five specific questions you should answer to determine if oneM2M is the correct choice for your application. Answering "yes" to 3 of these questions would indicate that oneM2M is the correct choice for you.
- Do you know what "ICT" stands for, and is that you? (Information and Communications Technology)
- Is the cellular network your primary connection technology?
- Are your target applications largely composed of moving parts?
- Can the components of your system tolerate intermittent connections and loosely-controlled latencies?
- Will your system leverage services provided by a communications provider such as a telco?
Data Distribution Service (DDS)
The last framework that we will explore is DDS. I have to admit, as the author of this post and an RTI employee, I am biased towards DDS as I have been working with it for over 14 years. Of the four frameworks called out in the IIC IICF, DDS is the only one that provides a Peer-to-Peer, Publish/Subscribe architecture. With DDS, each application participates on a "databus" which creates a shared global dataspace. This means that the databus comprises a set of data topics, each defined with its unique data model, that is then discoverable by any participant on the databus. Once a peer application declares its intent to either publish data on a topic or subscribe to data on a topic, then through a discovery mechanism DDS will connect all of the appropriate publishers with their subscribers. Figure 5 is a diagram of a layered databus architecture that instantiates three databuses which connect through gateways located between the horizontal layers.
This example diagram is based on a healthcare patient monitoring application you would find in a hospital. However, this is just one example of the many types of real-time autonomy applications where DDS is used today. Other areas of applications include smart grid, Oil & Gas, autonomous drive vehicles, transportation and defense systems. Here are the five questions you should ask yourself about your application to see if DDS is the right choice for you.
- Are there severe consequences if offline for a few mins/secs/msecs?
- Have you said "millisecond" or "microsecond" in the last two weeks?
- Do you have more than 10 programmers?
- Does your data have many destinations?
- Are you building a next-generation IIoT design?
Wrapping Things Up
As I mentioned before, given that I work for a DDS company, I am partial to the DDS infrastructure and the problems that it solves for real-time autonomous systems. That said, I hope that this post gave you some tools to use to determine what is the right solution for your next or current distributed infrastructure project. Because, really, these transports and frameworks are all good at solving very different problems. The key is to figure out where your application requirements fit within the landscape of available solutions. The tools recommended here include the IIC Industrial Internet Connectivity Framework (IICF) and also the list of key questions for each solution. If there is something missing, please do not hesitate to reach out to me in the comments. I would love to keep the discussion going, and learn about any other solutions out there that can help developers and architects solve their connectivity problems without having to recreate the wheel with a custom proprietary solution.
- The RTI Blog >> Industrial Internet Connectivity Document Evaluates Core Standards: DDS, OPC-UA, WebServices
- Free On-Demand Webinar: How the IIC's Connectivity Framework Guides IIoT Connectivity Selection