The OT and ICS security community has unwarranted confidence in our ability to manage OT cyber risk. I write this as someone who has been working on this problem for 23 years now and has finally descended Mount Stupid and is making my way out of the Valley of Despair.
The community has coalesced around a set of good practices / cyber hygiene that they are highly confident is the right way to secure OT. Even though we have little or no evidence supporting the effectiveness of most of these good practices. We don’t even bother trying to measure the effectiveness of these good practices. Take it on faith or an unproven theory that they are the right things to do. Much like doctors followed unproven good practice by bleeding and inducing blisters on their sick patients.
Every couple of years I encounter the Dunning-Kruger Effect graph shown below. It resonated with me more than usual as I have come to personally realize the last three years that most of our OT security consensus good practices are at best unproven and many are likely wrong. The confidence we have as a community is strikingly similar to being on Mount Stupid in this graph.
And yet we are doubling and tripling down on this consensus. The community is called to implement an increasing long list of cyber hygiene from an increasing in number and vehemence organizations (government, industry groups, insurance, media). With little or no serious attempt to measure or otherwise test if this is the right approach to OT cyber risk management. Having asked leaders and practitioners this question the past three years I’m usually told it can’t be done.
We need to start measuring the effectiveness of the OT security controls to reduce likelihood and measures taken to reduce consequence. Executives need to hold security to task. No measurement of effectiveness. No money, resources, or deployments.
How much does it cost? What is your expectation (hypothesis) for the reduced the number and impact of incidents? What metric will you be reporting up to prove or disprove your hypothesis? While that data is being collected, how effective is it at reducing a simulation of the top five most common OT attacks?
- How much is the effort to have a near complete asset inventory reducing OT cyber risk?
- How about quarterly patching of OT cyber assets?
- OT detection solutions and information sharing?
- OT microsegmentation
- …
(Note: I left off OT/IT security perimeter, MFA for remote access, anti-virus / Windows endpoint protection because I believe these have shown their value. This is wrong. They should be on the list; everything should be measured.)
We, the OT cyber risk community, can no longer hide behind excuses that this can’t be done, or it is difficult. This is another facet of the Dunning-Kruger Effect that is right. Little competence is required to climb to Mount Stupid, and it takes the accumulation of a lot more competence (knowledge) to climb the Slope of Enlightenment. Time is not on this graph, and if Time replaced Confidence on the x-axis I believe the curve would look similar.
If you are looking for a glimmer of good news … we have a larger, more talented and more diverse group of people climbing to the peak of OT cyber risk management Mount Stupid faster than ever before.
The related bad news is that the summit is getting crowded. People are resistant to leave the summit and go down to the Valley of Despair and gain competence. Especially when the OT cyber hygiene view at the summit is so beautiful. So accepted as being right. Echoed as the right approach even by those who haven’t climbed to this summit.
It’s hard, and a blow to the ego, to shed confidence. To consider that what you recommended, while well-meaning and logical, is unproven and possibly wrong. To approach the problem with more exploration and experimentation and less certainty.
For OT cyber risk I’d make one small change to the graph. Add a small mountain, a false summit before Mount Stupid, that is named OT Is Different Than IT Peak. Almost everyone climbs this peak. It’s easy, and they write their article or give their presentation on this topic. Thankfully most don’t spend to long at that false summit.
I ran into this Dunning Kruger Effect chart most recently reading Dr. Peter Attia’s excellent new book Outlive. Peter is a guru in health, longevity and fitness to so many people. He had the humility to say he spent a lot of the last ten years hiking to Mount Stupid when it comes to nutrition.
He spent three years on a keto diet, tried all sorts of fasting regimes, and had the discipline to strictly adhere to a lot of ‘diets’. All based on a combination of rigorous study of scientific research and his personal experimentation. This led him to write and speak with confidence on the topic. Peter wrote “my maximal confidence and relatively minimal knowledge having propelled me quite close to the summit of Mount Stupid”.