Complex Event Processing – Making sense of all your data

So you have a distributed system and you’re happily sending data between nodes in your system. The consumer applications are consuming the data your producer applications are producing, and everything is running smoothly. Now, that doesn’t sound like any system you know does it? Distributed systems are by nature complex. Nodes and applications are not straight producers or consumers; they’re a bit of both. And there’s always some resource contention. This is where Complex Event Processing (CEP) comes on the scene. CEP allows you to run queries on streams of data in real-time, either transforming the data or triggering alerts based on data content. Let me explain by talking about a couple of use cases.

Let’s say you have a large distributed system where data is being exchanged between nodes and applications. You’re finding that some of your applications are getting starved for data, but everything looks fine in the network – there are only a few lost packets and plenty of bandwidth available. The next step is moving up the stack to see what’s going on with the networking middleware. Now, if you’re lucky enough to have a well-behaved middleware like RTI Data Distribution Service, there are ways to retrieve detailed statistics about the communication channels. Passing this information through a CEP engine you can write queries that determine when and where you have a slow producer or consumer in your system that is clogging everything up.

Another common use-case is event correlation. Seemingly disparate events may not be, and corroborating can trigger pattern based alerts. Anyone who’s watched a Hollywood spy movie knows this – increased cell-phone chatter coupled with certain keywords and the change in behavior of suspected operatives means something bad is likely going to happen. Using CEP events can be monitored and corroborated in real-time and alerts sent out when the situation warrants it.

CEP is an incredibly powerful technology, and with increasing data rates it is a technology that should be watched carefully in the next couple of years.

Learn More:

Autonomous Vehicle Production »

What is DDS? »

What is IIoT? »

Connext DDS Cert »