More Project Basecamp modules and tools have been released today. The Basecamp reaction has been predictable and disappointing at the same time. The initial furor is over the disclosure, and there continues to be very little anger over the fragility and insecurity that has been allowed to live in the critical infrastructure for more than a decade — and for the foreseeable future if nothing changes.
Almost all of the guidance from SCADA security guru’s has been focused on compensating security controls. The same controls that have been preached over the last five years, including on our site. It’s not wrong, but it’s not what is needed to finally make some progress of PLC security and robustness. Here are our recommendations:
SCADA and DCS Owner / Operators
New Projects
ICS have lifetimes measured in decades, so don’t make a huge mistake that will leave you with a fragile and insecure system for the next twenty years.
Stop. Demand the vendor provide you with a secure PLC/RTU or a clear upgrade path to a secure PLC/RTU. What is a secure PLC/RTU? Well there are a few sources for this such as our Field Device Protection Profile that we wrote under contract for NIST or the ISASecure Functional Security Assessment (at least level 2). We will have more on this later because it is an important question, but if you need something today you should focus on these two requirements:
- Authentication of the source and content for functions that could affect the process. This is required to provide process integrity and availability. Items like session initiation, changing ladder logic, changing firmware, stop cpu, change IP address, add user, change authorization, etc. need to authenticate the source and that the data has not been modified in transit before implementing the function or request. Add write commands to this list as a requirement and consider the performance impact of implementing this feature.
- Protocol stack robustness – A test similar to Achilles or the ISASecure EDSA to insure the PLC/RTU does not fail when scanned, fuzzed or receives unexpected data. This is required to provide process availability.
Very few PLC/RTU vendors offer this today, as seen in the Project Basecamp results. So you need to get the vendor committed to fixing protocol stack robustness issues and adding the authentication features. The upgrade path should be clearly defined in terms of when it will be available, the upgrade process (such as a software upgrade or replacing the Ethernet card), and the cost.
With this information the organization can decide if it can continue with the project or if it needs to be modified or even halted. I can hear the engineers and project managers out their groaning that this is unrealistic and would not be accepted by senior management. But this is a risk decision that you need to put in front of senior management. Let’s look at this with some Basecamp examples.
GE D20ME
This is an easy decision. The device is a dead end with 80’s/90’s technology that can’t be fixed. You are looking at a major upgrade and probably total replacement of the RTU to achieve even the basic availability and integrity requirements. Full stop on this project.
I’m not saying don’t buy from GE. Go back and ask them what else they have because you can’t deploy a fragile and insecure RTU with obsolete software and hardware on a new project with a planned lifetime of 20 years.
Rockwell Automation ControlLogix
A more interesting case because the product is potentially upgradeable. Time to have a forthright discussion with the vendor. When are they going to fix the Basecamp problems, prove protocol stack robustness, and offer source and content authentication?
This is not necessarily easy for the vendor because some of the issues, such as unauthenticated commands like CPU Stop, are part of the EtherNet/IP protocol under control of the ODVA (yet another post that needs to be written on delinquency of protocol organizations to begin adding security ten years after 9/11). There are solutions like tunnels and wrappers that can be vendor specific, but don’t try to solve the problem for the vendor. Instead, evaluate their solution.
Again, what is the timeframe, upgrade process, and cost when this basic PLC security will be available in the product you are planning to deploy? If there is not an acceptable answer to this question, stop.
Senior Management Decision
Put this information before the appropriate level of senior management and get their decision. Too often engineers and security professionals accept risk that they have no business accepting. This is not only wrong for the organization, but its a terrible career risk when something bad happens. Accepting this fragility and insecurity puts the entire process at risk. This is a huge financial, safety and company reputation risk. Who in your organization can say the company will accept that risk?
There’s no excuse anymore for not understanding the risk. If your project has a ControlLogix in the design, you now have a Metasploit module that will show how easy it is for an attacker with logical access to take the system offline. If you have a Koyo, GE D20 or Modicon Quantum you now have a Metasploit module that shows how easy it is for an attacker to gain the credentials that allow them to affect the integrity of the process in a dumb or intelligent manner. And Siemens and most, but not all, of the other field devices or controllers have the same problem.
Make sure that senior management understands that if an attacker is able to gain logical access to the field device they will take the controller and process down or worse modify it in a more clever manner (a la Stuxnet). And I would caution engineers and security professionals against professing that this access is not possible even if you are following defense in depth recommendations. There are too many examples and paths to gain logical access.
At the risk of repetition, how can an engineer, security professional or senior manager accept this level of fragility and insecurity in a new system for the next twenty years? Don’t do it. At a minimum demand a committed upgrade path before you lock in these problems.
Coming Up: Part II – Existing Projects and Vendors and Part III – Government and Standards Organizations
Hopefully loyal readers now accept that we need to address the decade old problem of insecure and fragile PLC’s/RTU’s/field devices, and the Basecamp information and tools provide some additional compelling evidence and demonstrations to prove the point to senior management.
So what should you do? Part 1 covered what Owner/Operators should do with new projects. Part 2 covers what Owner/Operators should do with deployed projects.
SCADA and DCS Owner / Operators
Deployed Projects
To be honest, the difficulty and cost of this problem is one of the main reasons that little progress has been made in PLC security over the last decade. These devices control and monitor processes that must run 24 x 7 x 365 so it is difficult to pull one out of service for an hour, day or week. And while servers and workstations in ICS have redundancy that helps with upgrades and patching, there is less redundancy deployed for field devices.
But let’s also be honest that PLC’s can be upgraded or replaced in a critical infrastructure SCADA or DCS. It doesn’t happen often, but there are PLC refresh projects going on right now in most sectors. Engineers know how to do this; it’s only a question of the facing the problem, prioritization, and willingness to spend the time and money to do it — if they can find a vendor who will offer a secure and robust replacement or upgrade.
When are you scheduled to replace your PLC’s?
If your PLC’s have a remaining lifetime of less than three years, you are probably already beginning a project to replace them and focus on that, see Part 1.
If your PLC’s have a remaining lifetime of 3 – 7 years (numbers picked a bit arbitrarily) you should consider accelerating the replacement program to 3 years, see Part 1. This is where senior management should earn their money and make the hard decision as to whether to accept the demonstrated risk or accelerate the replacement and absorb the costs of reducing the expected lifetime. The key here is for the engineer and security professional to not be a hero.
It’s natural to be a problem solver and say “if we put all these security controls, defense in depth, it will be harder for an attacker to reach these vulnerable and fragile PLC’s”. This is true, but be very clear that it only reduces risk and an attacker who can reach the PLC’s will affect the integrity and availability of the PLC with potentially catastrophic results. The defense in depth principle is being abused. It was never meant to result in ignoring the weak link in your security chain. The solution is to strengthen the weak link, not ignore it.
If your PLC’s have a remaining lifetime of seven years or more you should determine if the vendor has an acceptable upgrade plan you can implement in the next 3 years, see below.
Why the focus on three years? Sooner would of course be better, and if your organization can speed the process along than it should be done. However this is a major engineering project and there is risk to the process if the upgrades are done improperly. Engineers can handle this work, but it actually could increase risk if they do not have the time to perform the testing, phased deployment, roll back options, etc. It is fine if the organization says it will take one year, two years or four years rather than three, if the associated risk is accepted by senior management. The important point is to finally start addressing this problem.
What is the upgrade or replacement product?
Start with your existing vendor, if you are happy with their product. It’s time to find out what they are going to do to upgrade the product to a secure and robust PLC. As mentioned in Part 1, focus first on source and content authentication and protocol stack robustness testing, but also query them on their intention to meet the ISASecure or other embedded device security standards that include functional requirements, protocol stack robustness and the vendor’s security development lifecycle. The questions:
- When will you have a security upgrade to the PLC and what does that security upgrade provide?
- How is the security upgrade performed? (eg Ethernet board replacement or firmware upgrade)
- What is the cost?
If the answers are unavailable or unacceptable issue a Request For Information (RFI) to look for alternatives.
Owner/operators should also open up the PLC and look at where it stands in terms of hardware and software obsolescence. It may not make sense to upgrade an ancient system that will be even more obsolete five years from now; in these cases you should accelerate replacement.
Next the owner/operator should look at the level of internal effort to upgrade the PLC and compare that to replacing the PLC? The product cost of the PLC is often actually not the major cost in a deployment.
With all this information the owner/operator needs to determine if the existing product should be upgraded or replaced, and we recommend focusing on a maximum three year window to address the problem.
All the risk and cost information then must be placed in front of senior management. Get the clear decision on the record. One reason the PLC fragility and insecurity problem has not been addressed is senior management has been able to sidestep accepting the risk. Is a senior manager in an owner/operator going to allow a novice hacker or simple malware to bring the entire process down or an advanced attacker to perform a more sophisticated, Stuxnet type attack?
One last thought, Siemens marketing response to Stuxnet has greatly retarded progress. When Siemens lead engineers and C-level executives state that all Stuxnet vulnerabilities have been addressed, then C-level executives in owner/operator companies exhale. Your job as an engineer and security professional is to correct this falsehood and let your C-level executives understand the risk.
If this all seems like common sense advice, it is.