Last week’s article highlighted a recent paper, Fact and Fiction: Demystifying the Myth of the 85% by Azrilyant, Sidun and Dolashvili, and focused on the fact that 85% of the US population is served by public sector water companies, not the oft quoted 85% served by private industry.
Another issue brought forth in the paper’s data is we have over 100,000 small utilities serving a small population. For example, the paper identified a small number of large electric utilities that address 72% of the US population. This leaves a larger number of small utilities addressing small chunks of the remaining 27%. The paper also uses Florida as an example in the water sector:
- 5,198 water facilities in Florida
- 23% (~1,196) are public sector facilities service servicing 19.5M people (average service of 16,304 people per facility)
- 77% (~4,002) are privately owned/operated facilities servicing 1.7M people (average service of 424 people per facility)
While it is hard enough to secure large utilities with large facilities serving large populations, it is an even harder problem to get 100K small utilities to properly and effectively address OT cyber risk.
Many of these smaller utilities fall in an increasingly used term, the cyber poor. Josh Corman described cyber poor organizations as having at least one of the following:
- insufficient information / awareness
- insufficient incentives
- insufficient resources (and this third item is almost always present in the cyber poor)
What is to be done with a large number of small, cyber poor utilities where many have an OT security program, if it could even be called that, which could be easily compromised by an attacker?
This is a hard problem, and I don’t claim to have the solution in this article. I do believe that the three dimensions below could be part of a solution. A solution that many will not like, but is it better to continue to not address the issue?
Nationally: Make And Acknowledge The Hard Risk Decision Where To Place Resources
One of the hardest things for a politician or government administrator to do is tell a single person or a community that their critical infrastructure matters less. That it is not a priority, it will not get attention, and they may need to live with an outage of critical services.
Sorry small town, rural community, suburb that has kept its small municipal system. The federal and state government are not going to pay much attention to your OT or OT security. Yes, you may get some outsized attention if you have an incident, think Oldsmar water, and nothing else is happening. But under normal circumstances or if there is a nationwide or regional incident, you are on your own. From a risk management standpoint, it is foolish to act or pretend otherwise.
Yes, there will be plenty of guidance documentation and training on what should be done, but the resources will be placed on large utilities serving large populations. The exception may be a small utility serving a key facility.
I have lived this in the last year where my house had no potable water for almost three months due to a major storm, not a cyber incident. The island of Maui has about 140K people, our part of the system affected about 400 people and got the appropriate prioritized attention.
Small Utility: Must Assume Automation Fail … Live Like You Are Cyber Poor
The small utility can use automation to achieve some efficiencies. They just can’t rely on the integrity or availability of the automation to meet their mission to deliver safe and reliable (enough) services.
What if the Oldsmar water incident is a success story? An example of the way forward for small utilities rather than a tale of failure. Oldsmar’s OT security program had some glaring holes that were exploited by an attacker, but these failures did not put their customers at risk or result in an unacceptable, prolonged outage of a critical service. Oldsmar had some automation in the form of an ICS, but they also had multiple non-hackable controls in place, and the benefit of water being slow, to prevent a misuse or loss of the ICS from putting their customers in danger.
Small introduces resource limitations (cyber poor) but it also comes with the benefit that it requires less automation to deal with scale. The cyber poor can live with being cyber poor as long as they don’t deploy systems that require them to be cyber rich. Utilities need to live within their means.
All Utilities: Plan And Spend For Mission Critical OT As If It Is Mission Critical OT (or don’t deploy mission critical OT)
The ICS world is on its fourth decade of deploying OT without planning for and allocating the resources to maintain and support the OT. There is plenty of blame and history to go around to explain this. Vendors, asset owners, and consultants promoting the new technology, and asset owners wanting some great new technology. It makes it easier to sell and easier to buy if you can ignore the time and money required for the cyber maintenance and can double, triple or quadruple reasonable life cycle expectations.
My earliest encounter of this was when ICS computers moved to Windows and were deploying domain controllers. Default installs, no plan to periodically update, no trained Domain Admins. Vendors saying it’s easy tech, you don’t need to do anything, and call us if you have a problem. Most asset owners gladly accepting this position as they already were quite busy and had none of that cyber maintenance type of money.
This trend has continued with multiple new technologies, but I must say I have seen many large asset owners learning from past mistakes and budgeting for the cyber maintenance and training. As well as planning for more realistic life cycles. This experience and realism is uncommon in smaller utilities. The true cost of what they are considering would put what is being sold and what they want out of reach.
The third part of the solution would be to not accept being cyber poor if you have mission critical OT. You can’t buy and deploy mission critical OT and be cyber poor.
This may be the hardest part because who puts the brakes on the procurement of this great technology that will save us money and do all these great things … if we just ignore all these costs and responsibilities related to cyber security and cyber maintenance and the increased consequence of a successful attack.
——
I’m not confident this is the solution. It is the best I have at this point. The other approach to part three in this article requires outsourcing the OT to a third party who is not cyber poor and committed to addressing the cyber risk associated with Mission Critical OT. There will be an increase over current costs that ignore the issue, but it would be less than each small utility trying to do it on their own.
How would you address the small utility cyber risk problem?