I’m Mike Toecker, Computer Engineer. I’ve been working in the Electric Power industry for about 8 years now, doing cyber security and compliance work associated with the NERC CIP regulations. I’ve worked for a major electric power consulting engineering firm for 6 years, a utility for a year, and now I work for an ICS security company.
I’ve had the opportunity during my career to assess control systems in control centers, transmission substations, and generation plants all across North America (now internationally). As part of NERC CIP compliance, I’ve had to assess impact from loss of a computer system, and how that could affect bulk system reliability. I’ve had to dig deep into how these systems function and into network and system architecture of ~38 different installations.
So, when I say that Master Fuzzing and Bare Serial vulnerabilities, like the 25 that Crain and Sistrunk responsibly disclosed via ICS-CERT, concern me, you need to pay attention. The Crain/Sistrunk vulnerabilities are the first ones we know of to demonstrate attacks on two unknown/underassessed threat vectors:
- Compromising a Master from a Slave
- Attacks without use of more modern (read: IP based) communications
I’m not going to talk about these new threats in detail today, they require a lot of time to cover. In this post, I’m talking history of development regarding the NERC regulations.
In the original development of both the NERC CIP and the previous NERC 1200 regulations, serial channels were considered less risky than their IP-based counterparts. And rightly so, at the time. There was almost no awareness of computer security issues in the planning and deployment of these systems, and insecure practices were rampant. Most of these practices revolved around truly horrible design and implementation of IP based networks, which were becoming the norm in electric power. Additionally, development took place after and during some of the roughest Microsoft Windows vulnerabilities, those like Blaster, SQL-Slammer, Sasser, and others. So, the developers of the original NERC regulations focused on these high profile vulnerabilities and the threat vectors they would come in over.
This led to a regulation where network security was the main technical component, where emphasis was on identifying critical systems, putting those systems in a perimeter, and locking down that perimeter. This has been the direction for the past ~10 years.
But, as we track the development of exploits, vulnerabilities, and viruses after this, NERC has not made significant changes to this approach. As everyone on the ‘Corporate’ side tightened their perimeters, restricted services, and locked the internet out from their networks, attackers moved to new threat vectors. These new threat vectors were not based on network services and direct communication via the internet, they were based in email, browsers, and browser based components such as Flash, Java, and others.
We’re once again in a new world of threat vectors, but this time they are ICS based. Compromise of the industrial controllers for Stuxnet should have rung a bell in NERC regulation development, it didn’t. Industry still has no strong standards for ensuring that logic designed is the logic running. The Crain/Sistrunk vulnerabilities are similar, we have research showing that Master stations can be compromised from the Slaves as early as June. And a few months later, we get another NERC CIP regulation submitted to FERC that explicitly exempts serial based connections from any regulatory security measures.
I find myself saying this often: The Crain/Sistrunk vulns are not the big deal. The big deal is that they demonstrate a general case that bare serial based communications are vulnerable to the same types of vulnerabilities that exist in IP communications. It shouldn’t be a surprise, but apparently it is. The NERC regulations have measures to protect, and detect, on threats against the IP communication vulnerabilities, but we do nothing against bare serial.
Crain/Sistrunk also demonstrate a general case where communications originating from a high security zone (a control center) to a low security zone (a non-critical substation) can be modified to compromise the high security zone. This vector was considered low-risk (I even heard the word ‘impossible’ uttered at least once) at the time. There were various reasons for this, most centered around the simplicity of many SCADA protocols, the lack of bandwidth on these connections, etc. This low risk was translated into a blanket exemption, which has been adopted by the industry as law.
Well, things are different now. While we’ve been lucky over the past ~10 years, and no threat vectors have required updates to the core of the NERC regulations, the general cases presented by the Crain/Sistrunk vulnerabilities require industry to take a second look at all our assumptions, exemptions, and protections from the past. How would we react if this were a vulnerability in lean burning gas turbines that caused them to burn out under certain grid conditions? What about if verbal communications between operators were often garbled or confused, and this led to problems coordinating activities between different entities?
Normally, NERC would start a workgroup to study the problem and make appropriate recommendations for addition to the regulations. Why this isn’t done, I don’t know. Most of the development activities for NERC CIP come straight out of FERC, trying to meet requirements in NOPR documents. Maybe I’m talking to the wrong people, and should start talking to FERC and the PUCs.
title image by tombdmot