Success-Plan Services
Jan Van Bruaene
“Make sure you tell them about the software systems operating the lab,” my team told me as I was preparing to guide a tour of RTI’s IIoT Lab as part of the Silicon Valley edition of the Connext Conference. If you visit the lab, what stands out are rows and rows of hardware, from large servers to Raspberry Pi. Yet the physical systems in the lab only tell a part of the story of what goes on behind the curtain to build and test our software. The software systems transform the space from "a collection of computers" into a true lab. This coordination of lab resources by in-house software systems has been evolving and is integrated into the RTI internal build, test, and support processes..
Before we get into the details, let me paint a picture of the type of software development our lab needs to support. RTI Connext software runs on over 100 different processor/operating system/toolchain combinations, ranging from common Linux distributions on Intel processors, to niche real-time operating systems on custom embedded systems. We support various networking technologies and transports, and interface with a number of databases. RTI Connext libraries support development in C, C++, Java, C#, Ada, Lua, Python, Javascript, and Go. Our product suite consists of over 20 individual products. We continue to support and ship patches to older software versions, going back over ten years, due to the nature and upgrade cycles of customers’ IIoT systems. To support this, we have built an impressive IIoT lab.
At RTI we host and operate our own build and test systems. While some parts can easily be migrated to a hosted Atlassian, Amazon Web Services or Microsoft Azure environment, a vast part of the lab can not. Let's take a peek.
The RTI IIoT lab consists of three large groups of systems:
- The build and test cluster - running various x86/x64 virtual machines
- The enterprise and embedded test target machines - a combination of special hardware, real-time operating systems and less commonly used systems
- The scalability and performance machines - a set of both powerful test machines and a large bank of Raspberry Pi boards.
The Build and Test Cluster
Every day, developers and the Continuous Integration system kick off numerous builds and test cycles on our build and test cluster. This system is built using openstack and ceph to virtualize the compute and storage environment. We currently use Atlassian Bamboo for Continuous Integration, and are in the process of migrating to Jenkins. More on this later.
On a weekly basis, we start a full build and regression test cycle across all our supported architectures. This involves the virtualized systems, as well as testing on the enterprise and embedded test targets.
Regression Test Dashboard
We currently use a set of homegrown scripts to automate the weekly build. These scripts start native and cross compilations, reset embedded boards, run tests on embedded and enterprise targets, collect the results and provide them in a dashboard for the developers to review. Making this work across all architectures is no small effort. Our platforms team is not only expert at porting our software to various operating systems, they are also familiar with the particularities of each embedded target and how to automate the testing.
The Enterprise and Embedded Test Target Machines
Not all our test targets can be virtualized on our build and test cluster. For example, we support AIX on IBM Power servers and Solaris on Sparc. We even have customers using openVMS on DEC Alpha systems. When one of our customers built a custom (and expensive) system using Cell processors, we got a few Playstation 3 game consoles and are running Linux on them as a less costly alternative. They continue to work like a charm. We also have a variety of MacOS versions running, and hardened Redhawk Linux systems from Concurrent.
Various Enterprise OS Targets in the RTI IIoT Lab
The majority of the test targets are embedded systems. Many of these are systems on a chip. We have one of most processor families in the lab: x86, PowerPC, MIPS, ARM (including Zynq and NVIDA Tegra X2), etc. We support many real-time operating systems, including VxWorks, VxWorks 653, Integrity, LynxOS, QNX, Nucleus, FreeRTOS, real-time Linux, DEOS and Intime.
In some cases, especially when there is no equivalent system available for us to purchase, our customers will provide us the system, operating system and board support package (BSP). In most cases, our platform team will integrate the system and build the embedded kernels. Often we are one of the first to try out new BSPs or even the brand new hardware.
Various Embedded Targets in the RTI IIoT Lab
The Scalability and Performance Machines
The third group of systems are the scalability and performance test machines. We have a set of powerful x64 systems, isolated from the rest of lab to measure the latency and throughput of our software with every build. We also use these systems to perform discovery and other scalability tests, using homegrown test frameworks. For example, with our “Polygraph” test framework, we can verify the discovery protocol at scale. It allows us to answer questions such as: Do all applications which need to discover each other, do so? What is the CPU, memory and bandwidth consumption look like during the discovery phase? How long does discovery take?
In addition, we built a small Raspberry Pi cluster allowing us to test the scalability of our software in various network topologies. As part of one of our research projects, the research team built a system to easily deploy and manage the test applications and results when using lots of machines.
Software System to deploy test applications on a large set of machines
Networking
We test our middleware with a variety of networking technologies and topologies. We have a combination of 10/100Mbps, Gigabit and 10 Gbit Ethernet networks. We have a very robust cable management system and have built a wireless testbed to validate the Transport Mobility feature, while switching to other WIFI networks. We have a small Infiniband set-up, and in the past have tested our middleware over a satellite link as well. We can simulate dropped packets or corrupted packets.
If you have an opportunity to visit the lab, you will discover that it is full of equipment but not of people. Through networked power switches and serial servers, RTI engineers all over the world can virtually access and have full control over every system in the lab.
The Next Generation Build System
Our current build and test infrastructure has served us well over the past several years. However, we have reached its potential. We are in the process of building a new build system allowing:
- The engineering team to be more productive, through faster builds and using more modern technologies (e.g., Cmake, Conan.io, Jfrog artifactory, Docker containers and Jenkins)
- The release team to do more build and test jobs in parallel
- Simpler machine coordination and reservation
The RTI IIoT Lab is at the heart of the action at RTI Headquarters, working 24x7, 365 days a year to ensure that the mission-critical, distributed systems of our customers – current and future – are well tested and continue to outperform. Looking forward, we’ll share more about the new build system in a future blog post.