I have always admired the comments of Michael Toecker on our site and elsewhere, and offered him the opportunity to write an occasional blog entry here when he has something to say. Here is the first of hopefully many from Michael.
Many asset owners in the energy sector are moving past the NERC CIP assessment stage, and implementing security controls that provide support to strict compliance and automation. There are several options when it comes to implementation, and the most popular is being the vendor packaged solution, followed closely by a custom designed option. Regardless of what option is chosen, racking servers and installing software is only part of the task. After that comes configuration and tuning, which is often just as time consuming as installation.
On-site activities are dependent upon the type of controls being installed, but there are some common areas. First; ALL cyber assets within a defined Electronic Security Perimeter need to be covered by the controls. Not covering a cyber asset means that a layer of security is missing, but it also means a gap in your compliance documentation and time spent after installation correcting deficiencies. A savvy auditor is going to check your automated reports against your list of cyber assets, if something is missing it is a quick find and a quick fine.
Second, incomplete or improperly configured controls can impact your ability to operate, so testing according to a mutually agreed upon test plan is absolutely necessary. Some examples I’ve seen in the field are: excessive logging causing slowdowns and other resource restriction issues, anti-virus/anti-malware locking out critical programs, and renamed/modified accounts keeping programs from running. None of these caused production issues and were part of our plan to isolate and solve, but it did require troubleshooting and correction that took time away from the installation. In the vast majority of these cases, extensive testing of the security controls on the factory floor or lab would have identified the problems, and had a fix ready for installation at the site.
Third, specify the configuration of security controls. Vendors are providing and installing security equipment, not taking over your entire CIP compliance effort. Unless they are specified to do so, important things like new user accounts, changing default passwords, changing of default accounts, patching of systems etc, will be either last priority or not completed. Which means more time spent after the fact getting everything working and compliant.
So what can you do to make validate that the vendor has fully installed your system according to specification? Audit them. Run through your own checklists to make sure all systems are accounted for in the controls. Run the reports that each tool gives you, and analyze the results for being up-to-date and timely timeliness and accuracy. If you have developed procedures, run through them and document accordingly, possibly updating those procedure according to your site-specific considerations. This is no different from normal due diligence activities in the electric power industry, such as verifying concrete specifications with a core sample or simulating a false overcurrent to check for the expected protection equipment response.
All these pieces will help to limit instances of rework and will help your personnel get up to speed on these new tools. At the end of the day, your personnel are the ones who will be in the position of maintaining this system going forward, give them a leg up on.