In software-defined real-time systems, small timing regressions can have outsized consequences. Because of this, RTI performs rigorous performance testing before each software release.
Most recently, while working on the Connext Micro 4.3.0 release, our automated performance testing identified a minor performance regression in the critical send/receive path. Trying to understand the cause of this performance change led us to endeavor to measure performance at the lowest possible level.
Conventional sampling profilers can highlight where time is spent but often lack the detail needed to pinpoint the source of timing regressions. Hardware-assisted execution tracing, such as Intel Processor Trace (Intel PT), enables capturing the processor's actual execution flow with low overhead. However, existing analysis tools don’t enable the analysis of the latency of specific critical functions at the necessary level of detail.
Leveraging AI, we created a tool to analyze this data and make it easier to detect, investigate, and resolve performance regressions.
The standard approach to identifying which parts of your software are using the most time is to profile it using stack sampling. With stack sampling, the profiler will interrupt your program at a fixed rate and record the call stack at that point. Then with statistical sampling over time, it’s possible to extrapolate from those snapshots approximately what percentage of time is being spent in each function
We tried several different tools for stack sampling, such as gprof, perf, and Intel VTune, but all of these had the same issue. The reported percentages were highly variable from run-to-run, making it impossible to identify precisely which call-paths were the root cause of the degradation. At this point, we had to take a step back and consider from a high-level whether or not stack sampling was the correct approach.
On the given testing platform, Connext Micro was capable of sending upwards of 500,000 samples per second. This meant that the window we were trying to profile was approximately just 2us. Then within that 2us, there could be 10s to 100s of individual function calls of which we were trying to find just the ones taking a little bit longer.
Standard profilers usually stack sample at a rate between 100 to 10,000 Hz. With certain tools you can push this to higher rates, but the overhead of the sampling itself starts to significantly impact the performance of your software. Even with statistical sampling averaging over millions of samples sent, it becomes obvious that sampling isn't the right tool for measuring function calls which take 100s of nanoseconds or less.
Researching the problem of measuring performance at this scale suggested the use of Intel Processor Trace (PT). PT is a feature built-in to the hardware of most modern Intel CPUs. Unlike stack sampling, it records the exact execution order of every instruction on the CPU with cycle-level timing accuracy. Because the tracing happens in the hardware itself, the overhead is extremely minimal…especially compared to trying to sample at an extremely high-rate.
The downside of PT is that it produces an extremely high-volume of data which is highly compressed. Recording just a few seconds of execution will produce gigabytes of data which then takes 10s of seconds just to decode. But the fidelity of the data is unmatched. From the data, we can reconstruct the exact execution trace and see exactly where time is being spent.
However all of these benefits remain theoretical without a tool to decode and analyze the raw data. The main tool available is `perf` which can reconstruct the same types of analysis which it does for stack sampling but from PT data, but it does so by simulating a stack sampling of the PT data. This simulated stack sampling has all of the same downsides of actual stack sampling, plus it takes an extremely long time.
We did not find a tool that easily met all of our requirements, such as:
Creating the perfect tool for doing the exact analysis we required was entirely enabled by the use of AI coding assistance. AI enabled a single engineer to design and implement the tool in one week. This tool would never have been implemented if not for AI – not because it wouldn’t be possible, but because the months of effort required to implement it would not be justifiable.
An initial prototype was created using Python, but processing the huge amounts of data proved to be much too slow in an interpreted language. The final tool was created using Rust, because of both its speed advantages as systems-level language and its strengths when working with AI. The powerful Rust compiler helped ensure that if the code compiled then it was free of many kinds of bugs that are easy to make in languages like C and C++.
With the exact requirements for what kind of analysis we wanted to do, we were able to quickly develop a tool to meet each one of them. It is still slow to recreate the call-graph from the raw PT data, but once processed, various types of analysis can be done quickly and efficiently.
The tool enabled us to directly compare the differences between different versions of Connext Micro. With cycle-level accuracy and precise call counts it became easy to see where the performance degradation was.
The results showed that the receive path was taking 16% more time than it had previously and that the primary cause was an increase in the time spent acquiring and releasing locks. It also showed the exact call-trace that was leading to the additional locking. We were then able to rework the locking in the critical path and enable a 5% reduction in the amount of time taken to receive a sample compared to previous Micro versions.
This is how RTI approaches performance engineering: measure precisely, validate hypotheses, and avoid drawing conclusions the data does not support. If you work on latency-sensitive systems, revisit any investigation where sampling felt directionally useful but never conclusive. If the problem lives at nanosecond scale, the analysis has to live there too.
This development experience also offers an encouraging data point for the state of AI-assisted systems programming: a novel, non-trivial tool written entirely under AI guidance, in a memory-safe systems language, in one week. As AI coding capabilities mature, the barrier to building specialized internal tooling for performance engineering – and other domains – will continue to fall.
Connext Micro 4.3.0 is now available with all of the performance improvements enabled by our low-level profiling plus many more improvements and new features.
About the author:
Joseph Holliday is a Senior Software Engineer at RTI, and he’s been with the company for five years. During that time he has worked with the Connext Micro team on various projects and initiatives, including several related to characterizing and understanding the performance of embedded systems. He has recently driven initiatives at RTI focused on furthering the use of Rust. He holds a Bachelor of Engineering degree in Computer Engineering from Vanderbilt University.