The recent water security incidents are leading to a number of efforts to address, or fix, “the problem”. Some are short term sprints, such as the 100-day Fragile Foundations Sprint or the six-month Project Watershed 250 in Texas. Others will take that long or longer to define and start.
All projects should have a testable hypothesis. If we do x, then we expect y to change in some manner. This requires to know the current state of y. We need to measure, have a metric, for the current state to know if our efforts resulted in the change we predicted and wanted.
We are wandering around in the dark in OT security and risk management. We know very little about the impact of various cyber hygiene on loss. If these projects are going to be funded and the OT security invest their time, then we should have a clear measure of how well they did or didn’t work.
All too often in these government, public, and community projects activity is considered the measure of success. We met this many times. Performed this incident response exercise. Wrote this guidance document. Assessed this many asset owner sites. Issued this many recommendations. This has and will continue to not solve the problem. Fail.
It’s time to turn on a flashlight and get some hard data to test these hypotheses. For example, the goal could be to decrease the number of OT systems that are accessible by anyone on the Internet. Sounds like a reasonable goal since this is the access that led to most of the water system breaches causing the furor. We need to know how many OT systems are accessible today (this data is available) and how many are accessible at the end of the project or at periodic reporting periods.
There is a danger in this approach. All the work done may not result in the intended change. This can be a reason not to measure before and after the action. Another reason is identifying what and how to measure is not always as straightforward as the Internet connected OT example.
The hard work is worth it, and the metrics are unlikely to be perfect. The Project Watershed 250 example could have a red team approach that attempted to breach the systems via a one or more common TTP and determine possible post exploit impact. The hypothesis could be that the project will reduce the impact by identifying security issues and implementing recommended measures to prevent the impact. We measure the likely impact of a set number of attack scenarios before and after the project.
You would want consistency and independence in the before and after testing. The team testing after the fixes should not be incentivized to find positive results. And the realization that not everything will be a success. If were honest, most of the public and community efforts to date have not been a success. If they were, we wouldn’t be in the current state.