Owners conducting a NERC Cyber Vulnerability Assessment have a requirement to annually verify ports and services. On Windows and Unix based systems, it is trivial and safe to pull a list of listening ports and the configured services thanks to commands like netstat, sc query, and others (you can even do it through Bandolier credentialed scanning!). But, network and automation devices often don’t have these commands readily available, and these devices tend to be the most sensitive to port scans (or owners are simply not willing to risk the scans due to unknowns).
So, because of the potential frailty of certain devices, coupled with the operational risk, owners choice is limited; Schedule an expensive outage to scan the devices, potentially interrupting the business of electric power? Or attempt to scan the devices online, take other risk mitigation measures, and hope for the best? As responsible stewards of bulk electric system reliability, the risk of the unknown more often outweighs the benefit.
I’d like to float another option, cloning the devices and running scans within a test lab. The objective is to identify network, system, and device configurations that could affect the output of the scans. Then, document the reasoning behind a scan of the clone being representative (or even equal) to a scan of the original, including any assumptions or other conditions.
Be aware, cloning the systems in a test lab can be more costly, and cost will multiply by the number of devices that must be cloned. While the level of effort to conduct the scan is the same, the activities involved in loading the device, ensuring it is a reasonable clone of the original, and then documenting the test will add hours on a per device basis. But, the risk reduction may be worth the additional cost.
Assurance that the cloned systems will match the production systems from a cyber security perspective requires both cyber security professionals as well as automation professionals. There are a many variables that need to be identified, and it takes that expertise to decide whether or not a variable has an effect on the compliance requirements. For example, cloning a Cisco switch would require the firmware level at minimum and preferably the same model physical hardware, as there are some differences between firmware revisions for different models of switch. It could also require the exact same configuration be applied too, but certain options wouldn’t affect the scan. As an example, would the final output of the scan be affected by the number of fiber vs. copper ports? Probably not.
While the Cisco is relatively straightforward, what about a PLC, Relay, or DCS controller? Here is where the automation expertise is needed, as a cyber professional may not know what is necessary to clone a automation device. For example, some PLCs will listen by default on Modbus, even if there are no Modbus points configured, while others require the Modbus interface be loaded via a module. Additionally, most automation vendors do not include port and service details in firmware change logs, which make it difficult to track differences between firmware revisions. These details are important, if I were an auditor I would want to review the work done for an approach like this.
The level of documentation for this process is going to be based upon each owner’s assessment of compliance risk weighed against the level of understanding the personnel have. One owner might have test reports only, while another might have a full methodology that explains how they developed the clones. At minimum, I would document the basic steps taken to develop clones, and concerns for specific devices, such as the Modbus example above.
As far as cost, try not to think of this approach as a wasted effort for compliance, there are dividends for this investment. First, set up the lab as you would an automation lab, and look for problems that occur during the scanning. Problems can be reported to the vendor, and the next generation of devices will be more resilient to scans. The knowledge gained can allow an owner a better understanding of the risk of scanning production systems, which can lead to cost savings during the next year, making them more efficient at compliance meeting compliance requirements. Then, you can make a donation of intellectual currency to the community via the many (successful?) information sharing programs, such as those run by NESCO.
I’d like to see vendor support in this area. Vendors have the dedicated test equipment already, they have unit tests that can identify misoperation, and they have the knowledgeable personnel to conduct this effort. But since they have a limited incentive to do this, maybe a better question is “$Vendor, what would I have to do to rent your test lab for a few weeks?”
title image by HJ Media Studios