Persisting data in a Real-Time distributed system
Written by Jens Pillgram-Larsen
June 26, 2009
Most of the distributed systems we deal with at RTI have performance constraints at their core. Either the system is pushing the limits of the available resources, or the action-reaction timing is critical for a given event. In other words the constraint might be on throughput or latency (or increasingly latency vs. throughput). In these kinds of systems persisting data is a real challenge. In many systems it is becoming a requirement that the distributed data is persisted. Take for example flight systems and automated trading systems, where persisting data is necessary to adhere to regulatory demands. At RTI we have set out to make persisting distributed data as minimally intrusive, performant and configurable as possible.
The simplest case of persisting data in a distributed system is recording the data to a file system. This can be done locally for each node, or centrally on a "recording" node. Each system setup solves different problems. Persisting local state on each node allows you to recover from a local crash by reading that state during recovery. Recording all the data on a centralized node allows you to query the entire set of data recorded (not just the subset any node would see) to find patterns or anomalies, or do a post-mortem on a system failure.
Locally each RTI Data Distribution Service publisher or subscriber keeps samples in queues. To allow for local persistence a mirror of these in-memory queues can be kept on disk. After a crash the publishers and subscribers can then be created with the same state as they had before the crash.
Centrally you can run RTI Recorder that can be configured to subscribe to interesting data (or all data for that matter) and write it to a file system. This data can then be queried either in real-time or during off-peak hours. As an aside RTI Data Distribution Service and RTI Recorder have countless flight hours on mission critical systems!
Connecting the Distributed System to an Enterprise Database
In some cases simply having the data on a file system is not enough — the full power of an enterprise database is needed. At RTI we have solved this by having RTI Real-Time Connect for ODBC enabled databases. Now you can use RTI Data Distribution Service to implement a distributed database, or utilize a database for your persistent storage needs. This allows you to use any tool or query you are familiar with to work with a distributed set of data. The possibilities are literally endless!
No matter what the use-case — recovery, logging, post-mortem analysis or database integration — persisting real-time distributed data is here to stay. The challenge is persisting the data with minimal intrusiveness on the real-time aspects of the system. At RTI we work hard to make that happen.