Connext Tools for IIoT System Development, Part 1

In Episode 25 of The Connext Podcast:

• [0:27] From power user to RTI engineer: How Ken went from RTI customer to shaping the software
• [4:20] Challenges in developing distributed systems vs. traditional areas of software development
• [9:22] Overview of Connext Tools: What they are and how they are used   
• [13:27] Using WireShark to debug distributed systems
• [14:50] Common debugging use cases and how Admin Console can help

Related Content:

• [Blog] Implementing Simple Introspection with Connext DDS in C++14
• [Datasheet] RTI Connext Tools
• [Webinar] Accelerate Distributed Systems Development using Connext Tools
• [Webpage] RTI Labs

Podcast Transcription:

Steven Onzo: Hello everybody and thanks for joining us for another episode of The Connext Podcast. Today we're talking to RTI Connext Tools technical lead, Ken Brophy. Ken will discuss how Connext Tools can provide deep visibility into your distributed systems. He'll also share common customer use cases where Connext Tools have eased the development life cycle from design to production. Ken, welcome to the program.

[0:27] From power user to RTI engineer: How Ken went from RTI customer to shaping the software

Lee Johnson: Thanks for sitting down, you're in a lot of ways the spirit of Connext Tools at RTI.

Ken Brophy: You're too kind.

Lee Johnson: So, I was really interested to pick your brain about a lot of topics that relate to tools as they apply to distributed systems development, from de-bugging and trouble shooting, all the way to systems in operations and how our tools apply to things like monitoring and tuning performance. So, we'll get there eventually but as a way to kick off, could you tell us what you do at RTI?

Ken Brophy: I am the Tools team lead, so I am kind of the technical lead but also the internal champion for advocation of tools on behalf of customers. So I'm a former RTI customer, myself.

Lee Johnson: And where did you come from?

Ken Brophy: I used to work at General Dynamics in Pittsfield, where I live, in Massachusetts, and we used DDS on a pretty big project, a new class of ship design - a  surface ship - and it was just terrific. DDS itself did everything we wanted it to do, but it didn't do it in a way that was user friendly. We spent a lot of time fighting some issues and stuff, we thought there would be an easier way around it. So, in fact, we created a number of tools or ancillary software around the main product that RTI had, to make our lives easier. So we had design tools, we had a UML system, and you could put your topics and your QoS profile in there.

We actually had XML-based QoS long before RTI had it, and we had a de-bugger tool that will allow you to see what was running and get information from it, log it and look for failed connections in the topic. We had all that before I even joined, and when I joined RTI, that was where my passion was, to bring that but make it available to RTI customers because it was all locked up within that company. We weren't a commercial software company, nobody is going to buy the General Dynamics DDS de-bugger, so I wanted to take that to RTI, if I could and make it available for everybody.

Lee Johnson: Great, so you were looking to solve a couple of problems, one making DDS more accessible, more consumable to a set of software developers, with whom you were working, as well as provide aids to de-bugging systems as you were developing them.

Ken Brophy: Exactly. We had some software we developed in-house at General Dynamics for the ship, but we had vendors that came in and they would bring their DDS software and sometimes things didn't work and it was like ‘where do we go to figure this out’? So that's where some of the ideas came from to ease that integration, to help us in development and tuning, and to log things so we could go back after and do an analysis of what happened during that use case.

Lee Johnson: So you were obviously successful in your quest to come to RTI, and you're now that engineering manager for our tools team, which is great.

[4:20] Challenges in developing distributed systems vs. traditional areas of software development

Lee Johnson: So let's talk a little bit about some of the underlying motivation for tools, as we're terming them. Distributed system development is a software development discipline, as we see it. How would you describe it as being different from "traditional" areas of software development? And what are some of the challenges that go along with developing distributed systems?

Ken Brophy: So I think distributed systems are quite a bit different than other disciplines in software because inevitably to solve really big problems, you either need to make really big computers, which is not practical, or you need to spread the workload around to a number of smaller computers. And that spreading of the workload and the coordination of solving that bigger problem, I thought was always interesting. It's a non-trivial problem to solve because there are technical factors, how much data to send vs. how big do we make a packet or what protocols do we use and all that sort of thing. Yet it's really almost a human engineering problem too because you have to get people with different viewpoints, "We should do x," "We should do y", they have to be able to talk to each other because software has to talk to each other and so, fundamentally, you have a technical issue and you have a people issue to solve and get people to communicate.

Lee Johnson: Absolutely. And just to put some context to the size, the magnitude of these systems, can you speak to the types of systems and the number of nodes or participants in these systems that you've dealt with earlier in your career and working with customers now that you're with RTI?

Ken Brophy: Yes, so back at my former work, I can't divulge any company secrets, that sort of thing, but you know it was a fairly sizable system, you're talking about a 300-foot ship. So there's plenty of human beings that have to run the ship and they, of course, need the computers to run systems, to run defensive systems, to run everything to email and everything else on the ship. They were fairly good sized at that point, but when I came to RTI, I got exposed to much bigger customer systems and consequently bigger challenges to solve in scaling up.

My first couple of years here we worked with a customer who still runs the biggest DDS system that I've ever seen or heard of and it's over a couple of hundred computers and thousands of participants. It's a big system, and we spent a significant amount of effort getting them to be successful, just due to the sheer scale and challenges that they had to overcome.

Lee Johnson: Certainly, and during the course of developing a system of that size, can you give some examples of types of problems, common scenarios that users would run into in de-bugging and trouble shooting behavior?

Ken Brophy: Right, it really varies but as a general rule of thumb, every time you increase the system size and order of magnitude, you run into new and interesting challenges to solve. So, most often the standard three bottlenecks are going to bite you in some way or another: memory, CPU and throughput. That tends to be the limiting factor of any computing system. Having in mind the end goal of where your system is going to be when you start designing, you can really avoid some of that. Using the products we provide, we try to help people locate those challenges, those issues, earlier on than they might have thought, that's one of the things we really like to do is to provide the information and let them be able to see what's happening.

[9:22] Overview of Connext Tools: What they are and how they are used

Lee Johnson: Excellent, so I don't think we can go too much longer without talking more specifically about the Connext Tools offering. What are Connext Tools?

Ken Brophy: Well it's a suite of products that help customers de-bug, test, design and get their systems functional and in their best shape to meet their requirements.

Lee Johnson: And give some specific examples, I don't know if we'll have time to go through every single bit of detail here but we can certainly talk to the more commonly used tools that our customers take advantage of.

Ken Brophy: Right, and it really depends on how you define a tool, there are products that my team produces, but there's also the services we offer the routing service, the recording service...

Lee Johnson: The runtime service

Ken Brophy: The runtime services exactly and so my team specifically, though we do things like the Shapes Demo, which is a learning tool to learn the concepts of DDS through an interactive application. The RTI Monitor, which is kind of a deep system de-bugger for tuning and system awareness. And then the Administration Console which does the distributed de-bugging administration and some monitoring of your DDS systems.

Lee Johnson: So what do you mean by administration, in this context, why is it important, why does it need a special tool to administer the services and everything going on in a DDS system?

Ken Brophy: Well it provides some commanding control for the RTI services, primarily. But also of our distributed logger component, and we do ship command-line administration shelves for each of our services but this is a UI. The Admin Console is a graphical interface so I think it's easier to work with, and each service has its own peculiarities so you can use the UI and not have to write a bunch of code yourself to interact with the service. It just saves people time and energy.

Lee Johnson: Now you mentioned a handful of tools, and I know there are a number more, but are there notable de-bugging flows where you see customers use it or that we prescribe to customers and how they apply our tools to solve particular types of problems?

Ken Brophy: Yes we have a variety of de-bugging offerings and depending on what you are trying to do and where, you might use one vs. another. So we have what we call these utilities and there's Ping, DDS Ping, DDS Spy, those are our two command-line offerings that really help you de-bug and identify problems in a network at the DDS level. Both of those tools will run on your target environments so if you have an embedded system, you can bring up Spy and see what data is actually arriving at that node. You can bring up Ping and see if DDS is able to connect to the rest of the network. Maybe your multicast isn't working right so you bring up Ping, you can see if Ping is connecting to Ping on other machines where you're developing so you can identify where there's a disconnect in the network connectivity.

[13:27] Using WireShark to debug distributed systems

Lee Johnson: At some point we have to bring up WireShark too and how WireShark is a member of this mix of tools.

Ken Brophy: Absolutely, WireShark is a critical tool for de-bugging distributed systems. It's a well known tool, it's got a lot of hours invested in it to make it a very stable and efficient tool, and we provide a dissector for our protocol to see the actual packets and to see inside them, what's contained in each protocol packet. WireShark, to me, is something that you really can't replace if you're de-bugging at that level, at the protocol level. What we try to do with the graphical tools is provide an easier-to-use alternative that you can de-bug many of the same problems but do so without having to write queries and write filters and things like that, and WireShark is a much more approachable interface to work with vs having to de-bug at a much lower level.

[14:50] Common debugging use cases and how Admin Console can help

Lee Johnson: So let's dig into more of the common de-bugging use cases that you see because these are cases that apply at a level much higher, typically, than de-bugging at the protocol level. Can you talk through a few of the core use cases for which Admin Console was used?

Ken Brophy: Sure. One of the big things that we see customers do with Admin Console is to de-bug QoS mismatches. The DDS API has a lot of quality of service settings, QoS, or "Quas" depending on who you're talking to, and there are a number of them which are involved in a contract between the producer of data and the consumer of data and you have to have an agreement on that contract, otherwise there's no communication, there's no data flow. And Admin Console does an analysis of the data it finds in your network to highlight those sort of mismatches and bring them to the forefront so you don't have to wonder what's happening - you can pull it up and just see that it's broken very quickly.

Another very common scenario that we see is that Admin Console can be used to visualize the data that's flowing in your distributed system. If you are suspecting that a particular topic has data on it that doesn't make sense or might be in some way corrupted or erroneous, you can subscribe to that data and bring it up in different visualizations to de-bug what's actually going on and identify the source of the problem.

Lee Johnson: And by visualizations you mean being able to actually see the data in real time, either in tabular format or in graphical format to give those quick queues to a developer about what is going on with their system.

Ken Brophy: Exactly.

Steven Onzo: Ken, thank you very much, that was some terrific information. And thanks to all you listeners for tuning in to another episode of the Connext Podcast. Next week we'll continue our conversation with Ken and explore the world of administration and monitoring of DDS systems. If you have suggestions or feedback on this or other episodes, please contact us at podcast@RTI.com. Thanks, and have a great day!