Governments have greatly increased the activity level on addressing OT security. Unfortunately, much, if not most, of the activity is wasted. I’ll use the US Dept of Energy’s CESER program as the example. This recipe applies to all government organizations, US and abroad, tackling the OT cyber risk issue.
STOP! … doing what private industry can and is doing better
So much of what .gov is doing in OT security is unnecessary, duplicative and usually much worse than what is being done already in private industry. This is muttered in private, and rarely said out loud due to fear of antagonizing government agencies. I brought this up in my interview with CESER Director Kumar, and I was encouraged to hear Rob Lee highlight it in his opening comments at the Senate hearing sitting next to Dir. Kumar last week.
Examples:
CyTRICS: Why is CESER funding vulnerability testing products from GE, Hitachi, Schneider Electric and SEL? These companies have internal staff that can do this. They have money to hire outside companies who meet or exceed National Lab talent. All for a lower cost and without the bureaucratic overhead. Director Kumar highlighted that the testing has already found vulns. The published vulns are embarrassing simple. Testing of GE UR family of relaysfound old versions of SSH, http for management, unauthenticated firmware upload, hard coded credentials. Testing of Hitachi Energy IED found credentials stored in plain text in backup files. Things any moderately skilled assessor would find in days. Hardly warranting an expensive, inefficient government program.
Funding of NRECA Essence Product: Why is DoE funding a product in an already crowded and commercially successful field of solutions (Armis, Claroty, Dragos, Nozomi and many others). The same could be true for large parts of the CRISP program.
Clean Energy Cybersecurity Accelerator: The first cohort showed secure remote access … an area with a large number of existing commercial solutions. The second cohort was to identify connected assets / asset inventory. Again a well established existing market.
Guidance Documents: I’d argue that much of the deluge of US Government guidance documents on OT cybersecurity are unnecessary. You could make the bully pulpit argument, but are we seeing this deluge have an affect? Which leads to:
Create Hypothesis and Metric
Most government programs have a vague hypothesis. Do this activity and it will somehow reduce risk, improve cyber hygiene, eliminate vulnerabilities. The hypothesis / reason for doing the program should be clear.
More importantly and almost always missing are one or more metrics for the program. These need to be metrics related to cyber risk, not activity metrics. The number of participants, the number of vulnerabilities identified, the number of advisories issued, the number of assessments done, are all activity metrics. Activity metrics measure if the program is being implemented, not wrong but insufficient.
The program needs at least one metric related to OT cyber risk to identify if the program is worth the resources. Is it reducing the frequency of critical infrastructure outages? Is it reducing the impact of critical infrastructure outages? Is the program degrading a threat actor?
Falsify Criteria
One of my favorite questions (stolen) to ask is: what is your favorite failure? Or what did you try that didn’t work and what did you learn from it? I asked Dir. Kumar this on the S4x23 stage and got a non-answer. He is far from alone in government in declaring every program is valuable and working well.
The lack of metrics, beyond activity metrics, is one reason government programs don’t fail.
We should identify what failure would look like before we begin. What would we need to observe to prove our hypothesis is false?
For example we are hearing great things about the detection and public/private information sharing progress over the last two years. And the great promise of the Energy Threat Analysis Center (ETAC) that is being stood up. Per the previous step, we should identify one or more metrics to identify success, again not activity metrics.
Equally important is to identify what would prove this effort is not reducing risk at a level to make the investment worthwhile. If we cannot identify a criteria that would falsify, prove wrong, the hypothesis and approach, then what we are doing is religion. Based on a never challenged belief.
We are assuming that sending this data and meta data around to a central site to correlate and analyze and distribute results will “work”. If we are not detecting, stopping and reducing impact of attacks at a level then it could be the wrong approach. There have already been years of USG detection programs and the drumbeat is we need more. I’m inclined to believe a hypothesis around this effort, and I believe it is unproven at this point.
Renew / Revise / Cancel
There needs to be an end time for these projects where the results are analyzed. The program may have worked as planned or even better and warrant more investment. The program may have failed, and a new approach and hypothesis pursued. Or we may learn that the program sounded good, but had little impact on OT cyber risk and should be cancelled. All three outcomes are valuable.
There is the temptation and tendency to say it didn’t work because we didn’t spend allocate enough resources. Don’t fall into this trap. It’s not a failure to say we tried something, and we proved it was not the right approach or solution.
CESER is pursuing efforts on ETAC and Cyber Informed Engineering that are non-duplicative of industry. The Senate asked some good questions on foreign content in the grid and prioritization information that could be worthwhile, only government should do this. These programs need a clearly stated hypothesis, one or more metrics, and a falsification criteria.