Real-Time Video Data Streaming

What This Example Does

This example can be separated into two modes of operation: Publisher and Subscriber.

Video Publisher

When in publisher mode, the main application will spawn a separate thread solely to receive the incoming video stream from FFMPEG over a UDP connection to the host machine’s loopback address, using port number 2277 (this number was selected arbitrarily and can be changed as needed in the code and command line).  

This approach was taken to mitigate the possibility of losing data packets over the incoming UDP connection, before they even reached the Connext publisher.  Unlike TCP, the UDP protocol has no packet-loss repair capability; any lost packets are unrecoverable.  The solution is to dedicate an entire thread of execution just to receiving these UDP packets from FFMPEG and moving them into a common memory buffer.   A shared mutex is then used to signal the main application that data is available for publication.

After initialization, the main application spends its time in a loop awaiting either:

• A signal from the shared mutex, after which it packs and publishes the collected video streaming data as one or more Connext samples.
• A ctrl-c (SIGINT) event from the user, which results in a cleanup and exit of the application.

Video Subscriber

When in subscriber mode, the application will create a Connext subscriber to the streaming video topic from the publisher, and will open a UDP writer to the host’s loopback address at port 2278.   When video streaming topic data is received by Connext, it is immediately written to the awaiting FFPLAY application over UDP.

With either application, configuration information is read from a local TOML file at startup to set the DDS domain number, sample buffer sizes, and endpoint station names (which are used to set the DDS data topic names).   Command line options are also supported.

Building the Example

• Clone the source code from our RTI Community GitHub repository
• Open a terminal in the newly-created directory
• Configure your build environment for RTI Connext
• Use CMake to create a build/make file set for your host environment
• Build and run the resulting application along with FFMPEG/FFPLAY.
  
  

mkdir build
cd build
"C:\Program Files\rti_connext_dds-6.1.0\resource\scripts\rtisetenv_x64Win64VS2017.bat"
cmake ..
msbuild cc_streamvid.sln

Linux Build Example

mkdir build
cd build
~/rti_connext_dds-6.1.0/resource/scripts/rtisetenv_x64Linux4gcc7.3.0.bash
cmake ..
make

Running the Example

Convenience scripts for launching the example have been provided for Windows and Linux environments, for 2 different configurations:

FFMPEG / FFPLAY STANDALONE TEST

The example can be run without RTI Connext by configuring FFMPEG and FFPLAY to talk directly to each other on a common PC machine. This test helps to verify correct codec operation and to establish a rough idea of end-to-end latency. In this configuration, FFMPEG and FFPLAY use the same UDP port number to directly communicate with each other, typically on the same machine (using UDP loopback address):

This configuration can be launched in one of 2 ways by running the script files:

scripts/run_demo_ff_only

This script will launch FFPLAY and FFMPEG to show the contents of the .mp4 video file included with this example.

scripts/run_cam_ff_only

This script uses an attached or integrated webcam on your host system as a video source – NOTE that the script may need to be customized to match your system’s camera configuration.

Testing with a camera provides an easy way to visually estimate the end-to-end latency of the video codec, which may be on the order of several seconds with default settings.

STREAMING OVER A NETWORK

Scripted

Script files have been provided to help automate the launching of the components used in this example: scripts/run_demo and scripts/run_cam_demo.  These scripts are similar to the above test scripts, but they include the Connext video streaming example application to send the video data over a network.

Command Line and Configuration Files

The example applications can be launched using a command line, with optional assistance from a local configuration file.   The priority order of the arguments is:

1. Specified on command line = top priority
2. Specified in configuration file
3. Default values

The arguments and configuration options are:

Notes:

• The ‘station names’ are used to create an 8-character hash ID name for the data topic.  This converts names like “MyLocalStation” into topic names like “MyLEmHus”
• The publication buffer size will have a direct impact on the publication rate.  See the ‘Tuning For Performance’ section for more information.
• The QoS Profile will select one of several profiles contained in the included USER_QOS_PROFILES.xml file.   Users are welcome to modify the provided profiles, or add new profiles of their own using the same naming convention of “profile_” + an identifying letter or string.

Tuning for Performance

Video streaming performance can be tuned to fit your network conditions and achieve a balance between image quality and end-to-end latency.  The adjustments and considerations include:

DDS Sample Size

This example uses a data type with an adjustable payload size – but there is a tradeoff:

• Larger payloads per DDS sample will reduce the number of samples to be sent and will improve network efficiency by sending more data payload per packet overhead.
  The drawbacks to larger samples include an increase in the end-to-end latency of the video stream: video packets must wait until an entire sample is full before sending. A larger sample size will also be more costly to video quality in the event of any samples lost during transmission.  Furthermore, if streaming video over the Internet, keep in mind that the MTU (maximum transmission unit) of the Internet is 1500 bytes; DDS samples that exceed that size will be broken into fragments, and the loss of any fragment will result in the loss of the entire DDS sample.
• Smaller payloads per DDS sample can result in very low end-to-end latency, because the encoded video packets will spend very little time awaiting an accumulation of enough packets to fill the sample before publication.   Sample sizes of less than 1500 bytes will fit within the MTU of an internet packet and can be sent without fragmentation over the internet, and the loss of individual samples will have a smaller effect on video quality than will a larger sample size payload.
  Smaller payloads can have drawbacks when the video streaming bandwidth requirement is higher.  For example, a 12mbit per second video stream would produce almost 8000 MPEG-TS packets per second; seven of these 188-byte packets could fit within a 1500-byte Internet MTU, which would require a publication rate of more than 1100 samples per second.
  
  

Video Quality and Data Rate

FFMPEG has command-line options to reduce the frame rate and/or picture quality produced by the h.264 encoder, resulting in a lower data rate of the video stream, reducing the burden on the network.   Please consult the command line reference for the specific version of FFMPEG you’re using.   As of this writing, the FFMPEG command line options were:

Reducing the Frame Rate:

-filter:v (filter the video only)

-fps=10 (limit output rate to 10 frames per second)

These settings will reduce the video frame rate only, without affecting any audio data.

Reducing the Data Rate:

-b:v 250k (limit encoder’s output bitrate to 250kbps)

-maxrate 250k (limit output data rate to 250kbps)

-bufsize 120k (sets the size of the output data buffer)

These settings will affect image quality by increasing the compression

Quality of Service (QoS) Settings

This example provides 2 example configurations for QoS:

1. BestEffort: samples are published without using the DDS reliability mechanism, seeking to minimize latency by eliminating the possibility of delivery delays caused by re-publishing samples that have been lost during network traversal.
   
   
2. Reliable w/Asynchronous Publication: samples are published with DDS reliability.  This option has further tuning to adjust the heartbeat_period to minimize latency.

Note that the full breadth of Connext QoS options are available for this application.

When you have completed the Video Streaming Case + Code example, please visit the RTI Community to share your feedback, ask questions and show the world what you built with Connext. 

The Video Streaming project repository for this example can be found here in GitHub.