Using RTI Support for Troubleshooting DDS Systems

In episode 34, we speak with Belen Cara, Senior Support Manager for RTI. Belen and her team help customers quickly troubleshoot and solve technical issues related to their mission-critical systems.

In Episode 34 of The Connext Podcast:

Podcast Transcription

Steven Onzo: Hello everybody and welcome to another episode of The Connext Podcast. I'm Steven Onzo, your host. And I'm here today with Belen Cara, Senior Support Manager for RTI. Belen and her team help customers quickly troubleshoot and solve technical issues related to their mission-critical systems. Belen, thanks for joining us today.

Belen Cara: Thank you for inviting me.

[0:31] How RTI support helps increase customer productivity

Steven Onzo: Absolutely. This is a topic that hasn't been covered on the podcast, so we're excited that you can come and chat and give us some insight. Let's just jump right in. We're here to talk about RTI support and the interaction with our customers. How is RTI making our customers more productive?

Belen Cara: When you think about our technology, it's amazing. It's powerful, right? And we allow super complex systems to communicate. We make sure data gets where it has to be when it has to be there. But obviously, there's no magic. You have to tune the system, you have to make sure things are working as you expect. When you think you have everything figured out, suddenly something doesn't work. As it was working yesterday, right?

And that's where my team come in, the support come in. When our users are not seeing what they expect, when the performance is not what they expect, when things are failing even, they reach out to us and we work closely with our customers and users, we investigate, we debug together, we test until we find the root cause and the solution for their problem. Everything goes back to working again.

[1:37] Elements of RTI support

Steven Onzo: Our customers are so unique that it only makes sense that we would need a very special team to support them. What makes RTI support so special?

Belen Cara: This is a good one because I get asked this all the time when I talk to people. Our support group is kind of unique. First because of the kind of customers we have. They're experienced and sophisticated engineers building critical IIoT systems, right? So the issues they run into are real world, complex, technical problems in real engineering positions.

So this is not, when you hear support you envision somebody with a headset picking up the phone, right? This is not what we do here. There are no trivial questions. There's no recipe book for solving them. So at the end, our team is built of world-class engineers. I work with an amazing team of engineers, and we are engineers helping engineers. So we really understand what they're going through. We always try to understand their pain, the pressure. They have to get the issue resolved because they have a system that is not working right. The need for technical sound answers and solutions.

We always say our values are trust, honesty, and empathy. We understand they're having a problem, and that's a very difficult situation. We understand what they're trying to achieve, and we earn their trust because they know we're not going to stop until we get it resolved.

And then what makes the team more special is the way every engineer in the team takes ownership of the overall success of the customers, not about the specific question. It's about the ultimate goal they're trying to achieve.

We don't measure how well our support does in terms of how fast you close a case or how fast you pass it to somebody else. What we look at, at the end of the day, is ‘was the customer able to achieve what they were looking for? Are things working for them?’ So our engineers, once they take a case, it's going to be their case until it is resolved.

Steven Onzo: And they own it.

Belen Cara: There is no passing around. Even if more people get involved, they are still owning that. And they own that relationship. After a while, we build this relationship with our customers.

[4:15] What a typical support case looks like

Steven Onzo: That certainly is unique. I want to steer the conversation to what this process looks like. Can you illustrate what the process is like for customers who have questions and kind of walk through that typical scenario?

Belen Cara: When we receive a question from a customer, the first thing we do is make sure we understand the whole scenario. It's not just the specific question they have. It's the overall thing, right? The bigger picture. We need to understand what they're trying to do. What they think they're trying to do. That is not always the same thing.

We have to understand the ultimate goal and all the pieces that come together because it's not just our product. Our product is embedded in a much bigger system. So if this is not clear in the original description, we always get back to them with more questions. They say, "Hey, you answer my questions with more questions." Yes, we do. Because we're trying to collect all these details to make sure we're understanding things right.

Steven Onzo: Yeah, let's make sure we're on the same page.

Belen Cara: Exactly. And this context is what allows us to act and speed up the final resolution. Then, once we understand the problem we always try to reproduce in house, in our lab. We build a scenario that is similar and we try to observe what they observe. If you get that done, you know you're gonna solve it. Because then it's you against the problem.

Unfortunately, that's not always possible. You can't take problem out of a big system and replicate it in a small one. And that's when our engineers have to get creative. We do a lot of out of the box thinking. We discuss a lot as a team. We're very noisy. And we brainstorm together, and when you're totally blocked we just discuss the problem. Have you thought about this? Have you tried this? Have you tried that? And at the end, those are our most powerful tools, our team. The collective knowledge.

Steven Onzo: Giving support to the support team.

Belen Cara: Exactly. I wouldn't say it better. Can I copy that?

Steven Onzo: Sure.

Belen Cara: When we can't reproduce, we analyze the information we have. We try to figure out the most likely cause. We build a theory, right? And with that theory, we think what tests our customer can run in the environment to confirm or deny our theory. Or what additional information we will need. Sometimes it's more logins. Sometimes it's traffic captures. Or whatever, right?

And then we pass those tests to the customer. They run them. They report back. There's a bit of back and forth, working together. And that's the most important piece here. We can't do this alone. We need them and we need to work with them so that we make progress until eventually, and we're pretty good at this, we find what was going on. It seemed impossible at the beginning, but usually we always find the cause.

[7:12] Alternative support options

Steven Onzo: So we've described what a typical scenario would look like for a customer who has a support package. What are the options for those without a support package?

Belen Cara: For customers that don't have a support package, there are still many resources they can use to either work on their own or collaborate with the community of RTI or DDS users. The center point for all of this is our community site, community.rti.com. And that's public. It's open to everybody. In there, the first thing they can find is our documentation. The user's manual, the API documentation. Next to that they have an amazing knowledge base where we document solutions to common issues. They can go search "what does this error message mean?" And we may have an article there explaining the error, explaining the solution. We have best practices, how-to's, descriptions on how to understand different things.

The Support team actually works actively on maintaining the knowledge base. We keep adding content. When we see the same question twice, we document it. And we do that because, as engineers we know the first thing an engineer is going to do is go online and see if they can figure out the problem by themselves. So we try to make this, as much as we can, available to them.

Steven Onzo: So several resources if you wanted to try and troubleshoot yourself.

Belen Cara: And then of course we also have a form where they can actually ask their question and get help from others in the community.

Steven Onzo: Excellent. Now, I want to talk about a support case that's fairly famous here at RTI. I heard that RTI support received a long distance support call from the International Space Station, perhaps one of the longest distance support calls you can get. Can you tell me a little bit about that?

Belen Cara: Yeah, sure! Actually it wasn't as cool as a phone call. I don't think they have phones out there. We did get a support case from the International Space Station. I remember the day. I was almost ready to go home and I opened my mail and I see this email like, "We're running some experiments. We're in the International Space Station, controlling a robot that is on earth actually." It was here, close by, where the robot was. Everything was a success. It was even in the news I think. But they got this login message that they didn't understand. And the email said something like, "Obviously you don't waste an astronaut's time on debugging." Which, yeah, I can see that. So they wanted our help to understand that message, see if there was a concern. Or how to solve it or remove it, right? They even sent us a picture of the screen that was showing, it couldn't be a screenshot. The picture taken with a phone of the screen, showing the login message. I still keep that picture somewhere.

Steven Onzo: Of course.

Belen Cara: So to me, that's really one of those, "Wow! This is cool. What I do is such a cool moment."

[10:22] Additional support resources

Steven Onzo: Absolutely. Unbelievable. Earlier we mentioned empathy as one of the support team's core values. To put ourselves in the customer's shoes for just a second, I want to talk about what resources our customers can be using to ensure they are maximizing their use of Connext DDS. What are some of the most popular resources?

Belen Cara: First, we have a lot of useful tools as part of the product suite to help analyze and understand what is going on in the system. We have from command line tools, very simple like Spy, RTIDDSSPY. It listens to information that is being transmitted, to tools that are more complete like Admin Console that shows everything that is going in the system from a logical point of view, from a physical point of view. Then you have communicating even the data flow, and the configuration. We use these tools a lot in the support team. They are a part of our daily work. We also try to educate our users on them because the more they use them, the more autonomous they can be and the more they can actually resolve on their own.

Steven Onzo: Right.

Belen Cara: And then aside from that, I think our documentation is truly amazing. I use it everyday. Sometimes we will debate, "Does this do this or that?" You read it carefully and find exactly what things you should be doing. It can be a bit overwhelming sometimes also. There's a ton of documentation.

Steven Onzo: But most of the time it's in there somewhere.

Belen Cara: Yeah, exactly. The trick is finding it. And for that we have also the knowledge base to help us navigate that a little. Quick answers to common things. We have them in the knowledge base to make sure people can find them more easily.

Steven Onzo: So, at this point, I wanna take the opportunity for you to offer some advice to the listeners who are interested in RTI support. What would you like customers to do before reaching out to support? Is there a checklist that customers should follow before reaching out to support or is every case different?

Belen Cara: Even though almost every case is unique, in a way there are patterns, right? There are things that they have in common. If you ask me what is the most common question, I will say, "My system doesn't communicate," which doesn't say anything at the end, right?

But there are always a few things that users can do and information they can gather before they reach out to us so that we can jump right into the problem, right?

As I said before, the context really is important here. We need to understand the ultimate goal they're pursuing. So for instance, when two systems are not communicating, we do have a series of very basic things to check like, do you have connectivity? Can you ping the other machine? If you can, then can you try, we do have an RTI specific ping utility. That's the same thing only using DDS. Try that, then we know, okay DDS can work. Then we have Spy and we see if the problem is in their machine and nobody can get any data or if the problem is in the actual application that is trying to receive that data. That way we can scope down where the issue is.

And then one thing we ask almost every single time is the quality of service. Having the quality of service provider they use gives us a lot of information about how things should be working.

Steven Onzo: How they've personalized their system?

Belen Cara: Exactly. Yeah.

Steven Onzo: Okay.

Belen Cara: So there's a lot of things they can configure. If we know their specific configuration, it helps us understand how the communication should be flowing. And then of course, a description, high level, of the system. How many entities are there? How many applications are they trying to communicate? Is the data they're transmitting big, small. Are they publishing fast, slow? All of that leads to totally different scenarios. So even for the same question, “my systems don't communicate.” There are a bunch of different situations. So we need to understand all of that context.

Steven Onzo: Right. By them answering these questions, it gets you closer and closer to finding out what the solution might be.

Belen Cara: Exactly.

Steven Onzo: Excellent. I have one last question for you. Are you hiring for the RTI support team?

Belen Cara: Yes. Our team is distributed. We have engineers in Spain. Actually I work out of the Spain office. We have engineers on the West Coast, in the East Coast of the US. And that helps us provide more timely responses. But even though we are distributed, we still work as a single team. I always say, "We're not two teams. We are one team in two places."

We are looking for more great engineers for our team. Maybe I’m going to take the opportunity to call out to those great engineers out there that are looking for a good challenge and that like to work in real world problems, to interact with other engineers and help them make sure things are working like they're supposed to work. If they're listening, feel free to reach out to us because we're looking for great people.

Steven Onzo: Fantastic, we love to see the support team grow. Belen, I wanna thank you very much for coming on the podcast and chatting with us, giving us some insight on RTI support. Thank you very much.

Belen Cara: Thank you. It was great.

Steven Onzo: Excellent, and to all the listeners, thank you and we'll see you next time.

INSIDE THE PLAYBOOK

Get the latest updates and insights from the RTI newsletter

Autonomous Systems

Customers

Products

Products

Services

Support

Developers

Resources

Company