Proposal: A small group in the ICS world develop a test to determine if a “machine” can be trained from only historian data to perform as good (indistinguishable) or better than a representative Operator.
In 1950 Alan Turing wrote an article on what he called “the imitation game” in an effort to answer the question “can machines think”. It was based on whether a human monitoring text communication between two parties could identify if either was a machine. Or in other words, was the machine indistinguishable from the human.
The buzzwords are flying in ICS these days with artificial intelligence (AI), machine learning (ML), and ICS in the cloud. Some instantiations are widely accepted such as using digital twins and machine learning for predictive maintenance and efficiency studies. Others, such as using machine learning for process variable anomaly detection, have been proven in examples if not yet deemed practical in general. However, when one suggests that Operators will be replaced with machines this decade, this is usually considered heresy.
When you look at the data available and the function, replacing Operators with machines is actually the easiest of the three tasks described in the above paragraph. Consider:
- Operators are trained to perform a consistent set of actions based on information provided to them. When you see this, do that. Operator training, whether in materials, simulators or side-by-side a skilled Operator, consists of this standard output/action based on a set of input seen on the HMI.
- A capable Historian has collected years, sometimes decades, of this input/status of the physical process, the subsequent Operator action, and the resulting change in the status of the physical process. This is the data needed to train the machine.
- When Operators run into a situation that does not make sense based on their experience or indicates there is likely a failure somewhere in the ICS or process, the Operators have a set of call outs to engineers or other staff. An Operator is not an engineer, and is not expected or asked to go beyond their skill set.
The last bullet is sometimes not true. There are some extraordinary Operators who know and can troubleshoot the ICS process as well as the engineers that designed it. This is why I used the adjective “representative” before Operator in the proposal. If the machine handles the drudgery of normal operations, this would free the extraordinary Operators to do the more valuable work of being Trainers, Maintainers and Sustainers to the AI/ML as described in the book Human + Machine.
The Operator Turing Test
There are a number of ways this test could be created. An asset owner or vendor could create and run the test on their own. Ideally it would be a pairing of an asset owner, to provide years of historical data and representative Operators, and a vendor, to help with the simulation and replay.
Once the test is constructed and tested with the Operators, then the historical data could be provided to any teams that wanted to take on the test. The actual test of the Operator and machine would not need to be run simultaneously as all we need are the results, the actions captured in the Historian.
Speed of action should not be a factor in comparing the Operator and machine. It is almost certain the machine will be faster. The judging or evaluation could look at:
- Did the machine take the same action as the Operator?
- Where they differed, was the machine action or Operator action superior?
- The number and necessity of the machine and Operator callouts for special situations.
The team developing the test would need to identify the factors that would measure performance in advance to determine “superior” and a scoring mechanism. Case by case analysis of variations is possible and may be worthwhile. How practical it is would be determined by the number of instances where the actions were different.
I’d be very interested in assisting anyone who wants to create this Operator Turing Test, and would love to be able to either provide details on the test or the results (can we move that fast?) at S4x22 next January.
Subscribe to my ICS Security: Friday News & Notes email.