Recording Service 5.1: Faster, More Scalable and More Concurrent than Ever!
Written by Juan Luis Jiménez Simón
March 16, 2014
You’re facing a problem in your DDS system: you want to use Recording Service 5.0 to record high throughput data coming in from sensor networks. The database has to be accessed by other applications while Recorder is continuously recording. Your topics are updated frequently and Recorder has to write at such speeds that it locks some of your other applications out of the database. When you open the database, you realize that although your types are small, the tables are big and full of columns you don’t really need. What can you do??
RTI Recording Service 5.1 was released a month ago with one of the biggest feature releases across the entire suite, including Recorder, Replay, Recording Console and Converter.
There were three motivations behind the updates to Recording Service 5.1 (RS):
The OMG XTypes Specification, which required integration with XTypes, including Mutability and types with optional members.
Enhanced scalability and performance, which resulted in additional configurability of the SampleInfo and metadata fields being recorded, and a reduction in new default column sets.
Improvements in data concurrency, especially in systems where the database is accessed by other applications while Recorder is recording.
Your types evolve? No problem. Recorder records them, Replay replays them
Systems evolve and as they evolve, so do their data and data types. Recorder and Replay support the XTypes specification, with mutable types and types with optional members. Recorder and Replay allow you to choose what version of a type to record or replay, by providing the types via XML. A new feature that easily allows mapping of topic names and type versions in the configurations.
Recorder and Replay administration types and topics are now also using X-Types. From this point forward (starting with 5.1), if the types evolve in the future, legacy applications will be able to administer the tools.
Configurability for enhanced scalability and performance
One of the coolest new features in Recorder is the ability for the user to select the SampleInfo and metadata fields to be included in user topic tables, as well as which fields to store in the discovery tables. Also now, by default, only a few extra fields are recorded apart from those of the user data types - just the necessary fields for Replay and Converter to work with the database file. See section 4.5.1 of Recording Service User's Manual for details and examples.
These new settings and feature enable a boost in the performance and scalability of Recorder. It can now record more user data fields and fewer irrelevant fields. Compared with version 5.0, the size of the database can be significantly reduced in cases where the user types are small- to medium-sized.
Boosting concurrent access to the database while Recorder is recording
There are new features that enable improved concurrent access to the database while it's being recorded. Highlights include:
- Ability to specify SQLite pragmas upon creation of the database file (and subsequent segments).
Among other things, this gives the ability to change SQLite from journal mode to WAL mode. Write-Ahead Logging offers improved concurrency because readers of the database don't block writers and a writer doesn't block readers.
- Online indexing.
Recorder can create and maintain indexes on the database while recording. Indexes can be established for any SampleInfo fields or for user data fields (when recording in deserialized format). When other applications need efficient access to certain tables, and the access is based on the content in certain fields in those tables, online indexing can really improve the overall performance of the system. However, remember there is a tradeoff: Recorder's performance will decrease as the number of indexes to maintain increases. Check out section 4.8.2 of our RTI Recording Service User's Manual 5.1 for tips and tricks for indexing in SQLite databases.
There are lots of new features, improvements and cool changes I haven't covered here, including automatic path separator detection in Replay, a new and faster deserialization algorithm for Converter and topic filters in QoS settings.
Check out the RTI Recording Service 5.1 Release Notes to get a sense of all this new stuff!