Introduction

Real-time streaming of video and audio data from remote locations is often a system requirement, but it can lead the developer down a path of complexity when trying to balance latency and image quality when running on a constrained or congested network.   

The technology behind streaming video codecs is continuously evolving; as of this writing, nearly 40 different video codecs are listed on Wikipedia.  However, they all share a common trait of converting the video and audio to and from a stream of data. RTI Connext is uniquely suitable for delivering that data in real time across local (LAN) or wide-area (Internet) networks.

This example illustrates the techniques, tuning and tradeoffs for streaming video data in real-time by focusing on the moving of data, leaving the video codec portion to external processes. In particular, this example is using the popular FFMPEG multi-codec video streaming application and its companion FFPLAY playback viewer application. FFMPEG offers a wide range of options to balance video quality against network capacity, and is available as a binary download or can be built from its source code. Note that other codecs / applications could be easily substituted for FFMPEG in this example.

This example is intentionally lean, with only a few source files to produce a single application that:

  • Has command-line options to act as a video data stream publisher or subscriber.
  • Has an external configuration file to set DDS sample sizes, domain numbers and endpoint identifiers.
  • Connects to FFMPEG / FFPLAY using a UDP connection through the host’s loopback address (127.0.0.1) and a designated port number (2277 or 2278).

This application defaults to using BestEffort reliability to achieve the lowest latency, however an additional QoS configuration example is provided using Reliable reliability, which may be a better choice when streaming high-quality video over a bandwidth-constrained network. for real-time video streaming, particularly when using a WAN connection such as the internet.   While reliable QoS is an option, the added latency incurred when repairing a dropped sample can directly affect the perception of real-time responsiveness, especially when used over the Internet.

This example has been built and tested on both Linux and Windows hosts to produce error-free transmission of 1080p high-definition video and audio over extended periods of time.   It has also been tested on constrained platforms and across constrained networks using the compression features of FFMPEG when needed to reduce the required network bandwidth.

What This Example Does

This example can be separated into two modes of operation: Publisher and Subscriber.

image2-Feb-09-2022-11-19-59-00-PM

Video Publisher

When in publisher mode, the main application will spawn a separate thread solely to receive the incoming video stream from FFMPEG over a UDP connection to the host machine’s loopback address, using port number 2277 (this number was selected arbitrarily and can be changed as needed in the code and command line).  

This approach was taken to mitigate the possibility of losing data packets over the incoming UDP connection, before they even reached the Connext publisher.  Unlike TCP, the UDP protocol has no packet-loss repair capability; any lost packets are unrecoverable.  The solution is to dedicate an entire thread of execution just to receiving these UDP packets from FFMPEG and moving them into a common memory buffer.   A shared mutex is then used to signal the main application that data is available for publication.

After initialization, the main application spends its time in a loop awaiting either:

  • A signal from the shared mutex, after which it packs and publishes the collected video streaming data as one or more Connext DDS samples.
  • A ctrl-c (SIGINT) event from the user, which results in a cleanup and exit of the application.

Video Subscriber

When in subscriber mode, the application will create a Connext DDS subscriber to the streaming video topic from the publisher, and will open a UDP writer to the host’s loopback address at port 2278.   When video streaming topic data is received by Connext, it is immediately written to the awaiting FFPLAY application over UDP.

With either application, configuration information is read from a local TOML file at startup to set the DDS domain number, sample buffer sizes, and endpoint station names (which are used to set the DDS data topic names).   Command line options are also supported.

Building the Example

To build and run this example on your network:

  • Clone the source code from our RTICommunity GitHub repository
  • Open a terminal in the newly-created directory
  • Configure your build environment for RTI Connext
  • Use CMake to create a build/make file set for your host environment
  • Build and run the resulting application along with FFMPEG/FFPLAY.

Windows Build Example
mkdir build
cd build
"C:\Program Files\rti_connext_dds-6.1.0\resource\scripts\rtisetenv_x64Win64VS2017.bat"
cmake ..
msbuild cc_streamvid.sln
Windows
Linux Build Example
mkdir build
cd build
~/rti_connext_dds-6.1.0/resource/scripts/rtisetenv_x64Linux4gcc7.3.0.bash
cmake ..
make
Linux

Running the Example

Convenience scripts for launching the example have been provided for Windows and Linux environments, for 2 different configurations:

FFMPEG / FFPLAY STANDALONE TEST

The example can be run without RTI Connext by configuring FFMPEG and FFPLAY to talk directly to each other on a common PC machine. This test helps to verify correct codec operation and to establish a rough idea of end-to-end latency. In this configuration, FFMPEG and FFPLAY use the same UDP port number to directly communicate with each other, typically on the same machine (using UDP loopback address):

image1-Feb-09-2022-11-19-58-98-PM

This configuration can be launched in one of 2 ways by running the script files:

scripts/run_demo_ff_only

This script will launch FFPLAY and FFMPEG to show the contents of the .mp4 video file included with this example.

scripts/run_cam_ff_only

This script uses an attached or integrated webcam on your host system as a video source – NOTE that the script may need to be customized to match your system’s camera configuration.

Testing with a camera provides an easy way to visually estimate the end-to-end latency of the video codec, which may be on the order of several seconds with default settings.

STREAMING OVER A NETWORK

image2-Feb-09-2022-11-19-59-00-PM

Scripted

Script files have been provided to help automate the launching of the components used in this example: scripts/run_demo and scripts/run_cam_demo.  These scripts are similar to the above test scripts, but they include the Connext video streaming example application to send the video data over a network.


Command Line and Configuration Files

The example applications can be launched using a command line, with optional assistance from a local configuration file.   The priority order of the arguments is:

  1. Specified on command line = top priority
  2. Specified in configuration file
  3. Default values

The arguments and configuration options are:

Command Line

Configuration File

Default Value

Purpose

-d / --domain

config.domainId

0

Sets the DDS domain number

-p / --pub

config.this_station_name

MyLocalStation

Sets the name of my publication topic

-s / --sub

config.from_station_name

FarEndStation

Sets the name of my subscription topic

-b / --buffer

config.data_sample_size

1316

Size in bytes of the publication buffer

-q / --qos

config.qos_profile

b

Selects the QoS profile

-v / --verbosity

n/a

EXCEPTION

Sets the verbosity of debug messages

 

Notes:

  • The ‘station names’ are used to create an 8-character hash ID name for the data topic.  This converts names like “MyLocalStation” into topic names like “MyLEmHus”
  • The publication buffer size will have a direct impact on the publication rate.  See the ‘Tuning For Performance’ section for more information.
  • The QoS Profile will select one of several profiles contained in the included USER_QOS_PROFILES.xml file.   Users are welcome to modify the provided profiles, or add new profiles of their own using the same naming convention of “profile_” + an identifying letter or string.

Tuning for Performance

Video streaming performance can be tuned to fit your network conditions and achieve a balance between image quality and end-to-end latency.  The adjustments and considerations include:

DDS Sample Size

This example uses a data type with an adjustable payload size – but there is a tradeoff:

  • Larger payloads per DDS sample will reduce the number of samples to be sent and will improve network efficiency by sending more data payload per packet overhead.
    The drawbacks to larger samples include an increase in the end-to-end latency of the video stream: video packets must wait until an entire sample is full before sending. A larger sample size will also be more costly to video quality in the event of any samples lost during transmission.  Furthermore, if streaming video over the Internet, keep in mind that the MTU (maximum transmission unit) of the Internet is 1500 bytes; DDS samples that exceed that size will be broken into fragments, and the loss of any fragment will result in the loss of the entire DDS sample.
  • Smaller payloads per DDS sample can result in very low end-to-end latency, because the encoded video packets will spend very little time awaiting an accumulation of enough packets to fill the sample before publication.   Sample sizes of less than 1500 bytes will fit within the MTU of an internet packet and can be sent without fragmentation over the internet, and the loss of individual samples will have a smaller effect on video quality than will a larger sample size payload.
    Smaller payloads can have drawbacks when the video streaming bandwidth requirement is higher.  For example, a 12mbit per second video stream would produce almost 8000 MPEG-TS packets per second; seven of these 188-byte packets could fit within a 1500-byte Internet MTU, which would require a publication rate of more than 1100 samples per second.

Video Quality and Data Rate

FFMPEG has command-line options to reduce the frame rate and/or picture quality produced by the h.264 encoder, resulting in a lower data rate of the video stream, reducing the burden on the network.   Please consult the command line reference for the specific version of FFMPEG you’re using.   As of this writing, the FFMPEG command line options were:

Reducing the Frame Rate:

-filter:v (filter the video only)

-fps=10 (limit output rate to 10 frames per second)

These settings will reduce the video frame rate only, without affecting any audio data.

Reducing the Data Rate:

-b:v 250k (limit encoder’s output bitrate to 250kbps)

-maxrate 250k (limit output data rate to 250kbps)

-bufsize 120k (sets the size of the output data buffer)

These settings will affect image quality by increasing the compression

Quality of Service (QoS) Settings

This example provides 2 example configurations for QoS:

    1. BestEffort: samples are published without using the DDS reliability mechanism, seeking to minimize latency by eliminating the possibility of delivery delays caused by re-publishing samples that have been lost during network traversal.

    2. Reliable w/Asynchronous Publication: samples are published with DDS reliability.  This option has further tuning to adjust the heartbeat_period to minimize latency.

Note that the full breadth of Connext QoS options are available for this application.

When you have completed the Video Streaming Case + Code example, please visit the RTI Community to share your feedback, ask questions and show the world what you built with Connext. 

The Video Streaming project repository for this example can be found here in GitHub.