Success-Plan Services
Akkshaj Singh
**Please note that with the April 2024 release of Connext Professional 7.3 LTS, the functionalities formerly comprising Connext Secure are now available as an optional component to Connext Professional, named Security Extensions.
The integration of video streaming with real-time data is emerging as a key use case for intelligent and connected systems across industries ranging from industrial automation to healthcare. In this blog post, we will discuss the benefits of using a data-centric framework for real-time video, and how it can be used for real-time monitoring, visualization and control of next-generation intelligent architectures and systems.
As shown in the chart below, surgical video data can benefit a large number of medical use cases, including real-time clinical guidance during a procedure, clinical workflow improvements, surgical procedure assessments, device usage and autonomous instrument control. The video is correlated with data captured during a surgical procedure that will enable more precise, effective and efficient procedures.
Figure 1: The metadata that would provide the most benefit if linked to surgical video data, as identified by survey participants, Source: SAGES Consensus Recommendations on Surgical Video, Figure 9, July 2023
Interoperability is critical to enable the necessary data streaming. The information flow of heterogeneous devices, data and video must be low-latency, reliable and scalable. Achieving this interoperability can be challenging.
There is a way, however, to achieve interoperability through the system middleware. The Data Distribution Service (DDS™) standard provides a data-centric communications middleware that rapidly passes information – regardless of source or technology platform – based on publish/subscribe permissions. Figure 2 shows how an intelligent digital system architecture may be decomposed into subsystems that have differing data connectivity requirements for video streaming, robotic control and monitoring. This enables the development of AI/ML applications that have seamless access to streaming data flow across disparate applications and subsystems, while also performing real-time processing at the edge.
Figure 2: Data centricity enables interoperability
Based on DDS, the RTI Connext® software connectivity framework is specifically designed to address the need for real-time data sharing in a distributed system. The Quality of Service (QoS) features of RTI Connext enable video streaming applications to be optimized for low-latency, high-fidelity video streaming, with advanced features such as filtering of samples to manage network bandwidth.
Of course, smart integration is crucial for maintaining video streaming performance, even when available bandwidth is unpredictable. To better streamline this integration process for customers, RTI offers a Case + Code example to build the plugin that allows seamless integration with the widely-used GStreamer media framework. The Real-Time Video Data Streaming plugin is now available, enabling Connext users to leverage GStreamer to easily configure other parts of their video application, such as resolution, frame rate and which codec to use (as shown in Figure 3). 
Figure 3: The video processing pipeline with RTI Connext and GStreamer.
We chose GStreamer for its rich community of plugins that all deal with various aspects of working with video, allowing developers to record to a file, measure latency or display the video feed to a custom UI. In addition, GStreamer, like Connext, supports a range of platforms. It can be used either directly from the command-line or with bindings in most major programming languages.
Along with open-source code for the plugin itself, RTI's Real-TIme Video Data Streaming Case + Code includes QoS profiles for different Connext transports, such as shared memory, UDP and RTI Real-Time WAN Transport. We have also included some sample GStreamer pipelines to get users started with this plug-and-play solution. Instead of spending hours tinkering with different video configuration options, users can begin streaming video over Connext in a matter of minutes and use the time saved to focus on the application itself.
With this plugin, video easily becomes a pivotal data payload that Connext can efficiently transmit alongside various other data types. This reaffirms Connext as the framework of choice for designing surgical and other safety-critical applications, as the video feed can be correlated and synchronized alongside the other data in the system.
While other solutions exist to stream video, their protocols (such as RTP) are limited to just audio and video and therefore cannot transport other data, such as sensor or control information. This latest plugin with Connext neatly solves that issue.
Running RTI Connext and GStreamer
RTI Connext meets the requirements for surgical video applications, due to its rich set of QoS options that allow developers to tune systems for optimal performance. It also supports different platforms and enables automatic discovery and low bandwidth consumption. Video requires special treatment, because transporting video comes with its own set of configuration options, such as choosing different resolutions, frame rates and codecs. Video data also tends to be quite large, so to meet low-latency requirements, it must use a different set of QoS tools when compared to smaller data samples, such as those typically used for sensors or control data.
The RTI Recording Service can help in this regard, acting as a “black box” that records all Connext traffic in the system in the exact order it happened. Security can be provided through Connext® Secure by configuring the QoS profile through XML, with no additional code changes required. This allows developers to use features in Connext Secure, such as authentication, encryption, access control and the logging of security events.
This Real-Time Video Data Streaming plugin is free to Connext users. For advanced use cases, RTI Professional Services also offers an Xcelerator (filter on Apply to find the Optimizing video streaming over Connext Xcelerator) to help developers tune the QoS configuration for each precise use case. As always, RTI is here to help!
About the author:
Akkshaj Singh is an Application Engineer at RTI with a focus on New Commercial Markets and Security. He has a background in robotics and embedded systems, and is excited to work at the forefront of enabling autonomy. He holds a BS in Electrical Engineering from the University of Washington, Seattle.