DNP3 Master Fuzzing

ICS vulnerabilities are easy to find and often not even necessary because the ICS applications and protocols are insecure by design. So why are the vulnerabilities that Adam Crain and Chris Sistrunk found in DNP3 protocol stacks such a big deal? Three reasons why I think this is the ICS security story of the year:

1. An attacker at a single, unmanned substation can crash an entire transmission SCADA system with a single packet (ok, maybe a few packets since there are likely redundant masters).

The protocol stacks of the PLC/controllers have been fuzzed, found lacking and are beginning to be patched. This is not news, and taking down a single substation is not news.

What the researchers have done is tested the DNP3 protocol stack in the master in the control center. They wait for the master to send a request packet, which occurs regularly, to the PLC/controller and then send back specially crafted response packets. Result: the master crashes. Impact: the control center loses the ability to monitor and control the SCADA network.

Note – they actually don’t even have to wait for a request packet since DNP3 supports unsolicited response.

When the master crashes it can no longer monitor or control any or all of the substations. There is no way to stop this with a firewall or other perimeter security device today. You have to let DNP3 responses through. In theory a future Tofino or Checkpoint application layer firewall that supported DNP3 could stop these malformed DNP3 response packets.

The researchers have found this to be true on almost all the systems they have tested which represent the big boys in the electric sector. By physically breaking in to 10 or 20 remote substations around the country they could bring down SCADA systems that monitor and control a large portion of the US power transmission and distribution systems.

2. This attack works with serial communications that are specifically out of scope of NERC CIP Cyber Security Regulations

The original version of DNP3 worked over serial, 4-20mA, RS-485 type communications. It still is widely deployed as a serial protocol in large systems, particularly in the less important (and typically less physically secure) substations or outstations. There is another version that encapsulates DNP3 in IP. Almost all attacks on ICS have focused on IP protocols.

The researchers have proven the same attacks that work on the IP version of DNP3 work on DNP3 serial.

Here is the kicker – the current NERC CIP cybersecurity regulations specifically exclude serial communications and the equipment that uses serial communication from meeting any security requirements. The researchers have proven the folly of this when serial communications can be used to take down the entire ICS monitoring and control capability.

I’m not arguing against IP networks getting most of the security attention, but serial comms coming into a control center are shown here to have a big impact.

3. This has been reported over six months ago and little or nothing has been done to address the problem in the electric grid

Adam proposed this research for S4x14 back in July. It was an easy yes, and in fact we have him teaching a class on Friday on response fuzzing and serial fuzzing. At that time he had been in touch with DNP3 Technical Committee, DHS/ICS-CERT and vendors for months. I have been waiting for all of these organizations, and NERC, to start pushing hard to get these master stations patched asap.

Instead, we have been hearing crickets. That’s not quite fair. ICS-CERT and vendors have been putting out very subtle, low key alerts and bulletins that there were vulnerabilities when patches became available. For example:

  • <ICS-CERT’s eterraControl Advisory: “Impact – Successful exploitation of this vulnerability could allow an attacker to affect the availability of the Alstom e-terracontrolsoftware. Impact to individual organizations depends on many factors that are unique to each organization. ICS‑CERT recommends that organizations evaluate the impact of this vulnerability based on their operational environment, architecture, and product implementation.” This is a far cry from saying if an attacker gains physical access to one of your substations he can stop your ability to monitor and control your transmission and distribution system.
  • DNP3 Technical Committee Announcement: Correctly states it is an implementation, not protocol issue but doesn’t raise any alarms. They also revert to the old excuses “SCADA protocols were designed for use on trusted networks. On untrusted networks, these protocols must be deployed within a system that uses adequate security measures.” and “No single security feature can defend against all types of attacks. Experts suggest using a defense-in-depth security methodology.” They never explain the impact of the vulns or encourage members to patch the vulns.
  • I don’t have access to the vendor bulletins; send them to me if you have examples of vendors emphasizing the impact and need to patch.

Why isn’t DHS, NERC, and the DNP3 technical committee telling vendors they need to fix this now and utility owners they need to get this patched asap? As much as I harp on insecure by design problems, this is a vuln that is actually much more serious. It is not that hard to gain physical access to a substation, especially one that is less important and still connected via serial comms.

It actually is a much easier fix than a PLC vuln because there are only a small number of masters, typically running on Windows or Unix Servers, that need to be patched. These systems are deployed with redundancy, and you could even stand up an additional server to help with the transition and possible rollback.

S4x14 Hype

Yes, we are very excited that Adam will be revealing the technical detail in public for the first time at S4x14. He thought January was the right time to give vendors and owner/operators time to address the problem, and the S4 audience was the best to understand the technical details. I’m most interested in the how they constructed the DNP3 response fuzzing packets that caused the crashes. I know it was not random data. The details on the categories of failures and vendor responses should also be very interesting.

Image by Chris Hunkeler