How PNNL and RTI Built a Secure Industrial Control System with Connext DDS
Written by Rajive Joshi and Erik Felt
December 4, 2019
The Pacific Northwest National Laboratory (PNNL) is one of the U.S. Department of Energy national laboratories, taking on some of the world’s greatest science and technology challenges. RTI, in collaboration with their cyber-security research team, created a proof of concept for a new secure solution for interconnecting commercial devices and applications used in the electrical power grid. The effort involved was formidable, but worth every minute!
The solution utilizes a new method for fully securing Distributed Network Protocol (DNP) 3.0 environments and enabling secure DNP3.0 data transmission at the individual data level. Using this approach, we were able to stop eavesdropping and man-in-the-middle attacks cold. Also, it raised alerts for various insider attacks, including identity spoofing by malicious parties. By contrast, legacy interconnections still in use by many today remain susceptible to eavesdropping, man-in-the-middle attacks and insider threats. Closing the door on this type of vulnerability is ground-breaking territory - and we couldn’t have done it without PNNL’s expertise and input.
Legacy SCADA Systems
Supervisory Control and Data Acquisition (SCADA) systems are used to monitor and control industrial equipment. Many industries deploy SCADA systems, including oil and gas, air traffic and railways, power generation and transmission, water management and manufacturing. In particular, electric utilities are major users. Which is no surprise, as scheduling, optimizing and controlling electric power generation and transmission require powerful, real-time SCADA systems.
As shown in the figure above, there are five components in a typical SCADA system:
- Field-based remote measurement and control equipment (called remote terminal units, or RTUs), which are used to manage the flow of power at an electrical substation. The substations are typically in remote locations, and usually unattended by workers.
- Programmable logic controllers (PLCs), relays or intelligent electronic devices (IEDs), which are used to automate tasks within a remote unattended substation.
- A control center with a set of central computers, which are used to manage the remote equipment located at various remote substations, and to process, analyze and archive the real-time data.
- A communication system, which is used to interconnect the substations to a control center.
- The operator interface (human-machine interface, or HMI) at a control center, which allows human operators to supervise and manage the flow of electric power.
Much of the critical national infrastructure using SCADA today is showing its age. These SCADA systems were designed for reliability and personnel safety; implicit trust of all components and communication was the norm, and the biggest security concern at that time was simply protecting device passwords. So, when initially deployed, SCADA systems did not really “need” to consider the threat from malicious intruders. But of course, times have changed a lot since then.
Today, with increased connectivity to the Internet and personal computers, the most prevalent threat is connecting to external networks through modern technologies like Ethernet and the Internet Protocol (IP). Although using these technologies makes systems functional and efficient, it unfortunately also opens our critical national infrastructure to cyber attacks.
Ultimately, these systems should adhere to the "Three Tenets of Cyber Security":
- Focus on critical data communication
- Control access to data
- Detect and defend against attacks
Applying these tenets to SCADA systems requires a framework that can securely communicate sensor information effectively in a real-time environment, connect this information to analysis technologies and also work seamlessly in unique SCADA environments.
Solving SCADA Challenges with DDS
RTI teamed up with PNNL to build a solution. Our design for the project leverages the open standards-based Data Distribution Service™ (DDS) message bus to securely interconnect devices and applications.
We created a rapid application development kit for DDS so the attack detectors could be quickly created and modified using the light-weight and extensible Lua scripting language. A distinct advantage of this approach is that now detectors can be added and modified without disrupting a running system or interfering with the critical data path.
We developed a data-centric "any-to-any" interoperability paradigm for interconnecting existing SCADA applications and devices that could be speaking a variety of other protocols. Our architecture allowed various SCADA protocols to be plugged into a secure DDS databus.
The solution architecture enabled rapid development and reconfiguration of attack detectors which could monitor and interpret the communications between devices and applications speaking various protocols. The architecture scaled horizontally, allowing new devices, applications and attack detectors to be added incrementally, without disrupting existing components.
For the demonstration at PNNL, we selected devices and applications that speak the DNP3 protocol, the most widely deployed protocol on the US electric power grid.
On the transmission substation, we deployed real-world devices (SEL 351A, SEL 451, GE L90) and on the control center we deployed HMI apps (Wonderware, Triangle Microworks SCADA Data Gateway) that are commonly used in the US electric power grid.
Implementing Security with Secure DDS
In the retrofit demonstration, we replaced the DNP3 link between the control center and the substation with a modern WAN network running an early access version of RTI Connext DDS with security extensions. Everything in the demo was compliant with the recently adopted DDS security specification.
The advantage of basing our solution on the DDS standard is that DDS works great over WAN. But because DDS is transport-agnostic, it’s not dependent on WAN. In fact, the solution is also designed to work seamlessly over private utility networks that might be using wireless mesh for automated metering. Particularly attractive is the fact that the solution works just as securely over the old leased-lines that are predominantly used across the United States, as well as over broadband and emerging cellular networks like 4G, LTE and etc0.
An effective DNP3 link was established between the devices in the transmission substation and the applications in the control center by tunneling DNP3 messages over the RTI Connext DDS databus. DDS security extensions allowed only authenticated components on the bus, and further allowed only components with "write" permissions to modify data and only components with "read" permissions to access data. The ScadaConverter unobtrusively exploded the DNP3 payloads into a data model suitable for analysis by the AnomalyDetector(s) written in Lua, leveraging the rich, structured data modeling capabilities of DDS for data in motion. The detectors continuously published metrics and raised alerts when anomalies were detected. This logical data flow is illustrated in the figure above.
Testing showed that our solution was able to securely tunnel DNP3 traffic between unmodified legacy SCADA devices and applications. We found that the secure DDS infrastructure did not negatively impact the operation of the testbed transmission substation environment. In addition, control system communication between client and slave devices was not interrupted.
At the same time, our solution was able to thwart eavesdropping and man-in-the-middle attacks. The data-centric attack detectors were able to tap into the flows and raise alerts for various insider attacks, including identity spoofing by a malicious component, unauthorized firmware upgrades, unsupported requests, high risk commands and denial of service (DNS) attacks. Without the solution, legacy SCADA communications would still be susceptible to both man-in-the-middle attacks and insider threats.
Our open standards-based extensible solution architecture - built on Connext DDS with security extensions - enables operators to monitor, analyze, visualize and respond to evolving attacks. It can leverage open-source and Commercial off-the-shelf (COTS) technologies, as well as new anomaly detection technologies. It can also collect security status and process it to detect attacks that target industrial control systems.
The solution lays the foundation for real-time detection capabilities, greatly increasing network reliability and defensibility of industrial control systems from cyber attacks. Overall, it's a timely, much-needed solution, given the current state of industrial control systems and just how much is at stake.
We’ll be highlighting these capabilities and more at DistribuTech 2020 in San Antonio, Texas in January. If you’re planning to attend, we encourage you to meet with us in the RTI booth and find out how we can help you take security risks off the table.
Want to learn more?
About the authors
Dr. Rajive Joshi is the Principal Solution Architect at Real-Time Innovations (RTI) working as the co-developer of RTI’s Connext product suite. He serves as Chair of the Industrial Internet Consortium (IIC) Connectivity Task Group and Automotive Task Group, focusing on accelerating IIoT technology. He is the lead author and editor of the Industrial Internet of Things (IIoT) Connectivity Framework (IICF) for which he won the IIC’s highest honor, the Individual Contributor Award in 2017.
Erik Felt is the Market Development Director for Future Grid at RTI where he is focused on bringing the benefits of IIoT standards and systems into the utility market. Erik joined RTI after spending seven years with GE Power (including five years with Alstom Grid prior to the GE acquisition) in the Software Solutions/Energy Connections business unit. His focus was on software solutions in the areas of SCADA, Energy Management Systems (EMS), Generation Management Systems (GMS) and Synchrophasor applications. Throughout his career, Erik has worked with utilities, generation companies and ISO/RTOs worldwide where the rapid changes in technology and the market's diverse needs required implementation across the utility spectrum. Early in his career, he worked in distribution engineering at two Midwestern utilities and in a consulting role on numerous automation projects for utilities across the Midwest.