How do you pick between 20+ ICS Detection and Asset Inventory solutions who are all claiming to be the best? The ICS Detection Challenge was designed to provide asset owner / potential customers with an unbiased technical comparison.
S4x19 ICS Detection Challenge As Planned
We were prepared for an awesome ICS Detection Challenge at S4x19, January 2019 in Miami South Beach. We had 400GB of ICS communication from a mining company. This included Rockwell/Allen Bradley, iFix, Modicon, OsiSoft PI, remote access, a DCS, mobile fleet (shovels, haulers, drillers), slope integrity systems and much more. The asset owner volunteer was extremely helpful. The asset owner also provided a detailed inventory, even including model numbers and software/firmware versions. I personally collected the packet captures (pcaps), and was confident we were in great shape for the Asset Inventory part of the competition.
Ron Brash of Deloitte did a huge amount of work (400+ hours) to anonymize the pcaps and generate and insert attacks into the pcaps. Some of the attacks were stand alone events, and some were related and part of a larger cyber incident. For the latter we were going to have the competitors show the judges how their system helped identify, correlate and analyze the larger cyber incident. Ron was assisted by Marc-Etienne Bergeron of Deloitte, Corey Thuen of Gravwell and individuals from Schneider Electric, Rockwell Automation and FireEye.
We had learned many lessons from the S4x18 Challenge and had cleaned up the scoring and other issues. It was going to be a great test in a condensed 8 hour time period that limited the analyst impact on the results.
S4x19 ICS Detection Challenge As Realized
You can skip down to the Lack of Participation section for more details, but in the end we had only Dragos, Kaspersky and an open source asset owner team willing to compete. Not enough for a competition. So we adjusted and provided a 130GB set of pcaps, that contained the multi-event cyber incident, to the remaining competitors. They received it late on a Friday and had until the next Wednesday to provide a score sheet and video showing their analysis using their system. These videos and information on the created attack were shown on the Main Stage as you can see below.
There was no asset inventory competition. There was no judging of the results. I do have a few thoughts from the Challenge.
Really Big Data (for real-time collection and processing)
One conclusion, that should have been obvious but wasn’t in the forefront of my mind, is that many solutions are going to gag on the ICS data. That is if they can even get all that data to a place where their analysis engines are. This 400GB was from one large mine, a subset of the ICS traffic that would ideally be monitored at that mine, and representing about 2 hours of collection per switch. The competitors referred to much of the data as “noise” a couple of times, but it is actually very chatty ICS communication.
The original plan was to feed the competitors 400GB over 8 hours to lessen the analyst impact on the results. There would not be much time for analysis. The results would be what the system automatically processed and identified. Dragos had purchased special hardware just to be able to ingest and store data at that rate.
Even with the scaled down data, 130GB, and extended time of 3 to 4 days to process, it was a multi-analyst effort to produce the videos you will see showing partial detection of the incident. This 130GB represented about 45 minutes of partial data from one mine. Imagine if you had a constant stream of this data coming from 10, 20 or even 100 sites.
Asset owners should be looking hard at the capacity issue. It tends not to be a focus during the trials and pilot. You should also be looking hard at the vendor’s distributed processing approach to their solution. Throwing out the “noise” early and close to where it is collected is important, likely essential in many cases, and only forwarding the alerts and a small percentage of the actual data. Then there is the related issue of the analysts dealing with all the events and alerts that this much data will generate.
My advice to asset owners is to go slow, for this and a variety of other reasons I’ll write up soon. Deploy your pilots, pick an initial winner. Deploy at a small number of sites and see how you deal with the volume of communication and alerts. And prepare management that you may not have picked your final product on this first deployment. It’s important you don’t get locked in politically to a solution that will prove to be impractical.
Detection and Understanding the Attack
One of the key items we wanted to test was the ability of the competitor, preferably in an automated manner by the product, to determine various events were related and understand the adversaries attack path, intention and impact on the ICS. We had included in the score sheet columns for relating events to previous and subsequent events, and it was going to be an area of emphasis in the judging.
The Kaspersky team did not tie the events together. They detected events in a more diverse group of areas in the ICS, but they did not show how these events were related. They were presented as five separate incidents. The Dragos team followed the putative attacker and showed his attack path. Given the additional time provided, it was unclear how much of this was due to analyst action vs the Dragos product. It did show well in the Dragos GUI as you can see in the video.
This may point to two different approaches for this category of products. The first would be to detect individual attacks and other cyber events on the ICS and send them to a SIEM in a SOC for analysis, a la Kaspersky. The second would be to actually analyze the attack on the product. This second approach would almost mandate the product take in security events from other sources, eg endpoint protection, firewall logs, Active Directory, … And it is consistent with Dragos’ view that there should be an OT SOC. Personally I’m not yet convinced an OT SOC is the right choice or even viable for most asset owners.
Difficulty of Testing Detection
There are a large and growing number of asset owner pilots occurring, relative to the market size and newness of the offerings. It is not particularly difficult, or dangerous in the case of passive solutions, to connect one of the products to your ICS and see how they do on asset inventory. Have your EWS or administration application run some diagnostic queries and you will get even more data. (Note: My current analysis points to asset inventory/management and detection being separate solutions that communicate with each other rather than the combined solution you see today.)
The question is how to test the detection? You can create a few one off events and see if the signatures catch them. You can add a computer to the network or new traffic patterns and see if the anomaly detection gets it. These basic tests aren’t going to differentiate the competitors, and they are far from what you would face if a moderately skilled attacker has access to ICS (the very reason you want a product in this category).
Asset owners could, in theory, hire their own team of Ron Brashs and spend two or more man-months to develop a rigorous test scenario. We are not seeing this, and quite frankly the solutions rarely encounter any serious adversary or attacks during the pilot or initial deployments. Right now winning is having the best marketing, sales, story and GUI. The later is very important and can be partially analyzed without a set of pcaps with a good cyber incident. The chance to test many solutions with the Challenge pcaps is sad lost opportunity to provide hard to get and useful information for potential customers.
Lack of Participation
Let me start with an appreciation of the difficulty and stress in being a startup in a fast moving and competitive sector. They don’t need my permission, but I understand that each startup should and will do what is their best interest, and they are under no obligation to be straightforward with why they chose not to compete in the Challenge.
Participating in the S4x18 or S4x19 ICS Detection Challenge took courage and confidence in your solution. If a vendor does poorly they will not only have to deal with potential customers seeing the results and the analysis and publicity that the Challenge team would generate, but also with the “winner” pushing a narrative beyond what the results showed (ahem, S4x18 winner Claroty).
To address that concern for the S4x19 Challenge we decided to rank the competitors by tiers (Top, Middle, Low) for asset inventory and detection. In the hope that this would prevent very similar results leading to mischaracterization. This did not help getting vendors to compete, nor did attempting to recruit every market participant I know of and many long email chains and conference calls.
We had two of the clear top four in the ICS Detection market, Claroty and Dragos, signed up for the S4x19 ICS Detection Challenge in October. Kaspersky, who we were late in inviting, jumped in as soon as their concern about getting visas was met. We expected that the other two in the top four, Nozomi and ForeScout (SecurityMatters), would return after competing and doing well at the S4x18 Challenge. And that we would have to actually limit the number of competitors we could allow to compete because many others would want to prove they were also Top Tier technically. It didn’t happen, and Claroty dropped out at the end of the year because they felt the risk of not performing well on some aspect would be used against them by competitors sitting out the Challenge.
You can draw your own conclusions why the ~25 vendors did not compete. Here are my opinions/analysis:
- The two others in the market leader category, Nozomi and ForeScout, felt they already were market leaders and didn’t feel there was upside. (There also may be some fear of the analysts at Dragos. Dragos’ product was not really in a position to be competitive in the Challenge at S4x18, in my opinion, but they have improved the product in the areas the Challenge tested and have hired a large portion of the highly talented analysts)
- Indegy has focused on Active components. Their Passive capabilities that would be tested in the Challenge may not be competitive.
- CyberX and a couple of other in the next level below the top 4 market leaders had enough concern they would not be competitive and are getting some pilots and some sales. They made the judgment competing was too risky.
- All the others … there are almost 20 vendors who are becoming afterthoughts in the Detection market … it baffles me why they would not compete. Even to be mentioned in the same breath as the market leaders is a win. Even the attention of competing would boost name recognition. There only risk in competing is potentially confirming what the market has already determined. But there is a chance that a couple of those 20 are technically strong and just lacked the marketing, sales, finance or infrastructure to succeed in the market. Given they all chose to pass on the opportunity my only conclusion can be that most knew they would not be competitive.
Future of the ICS Detection Challenge
Given the huge pro bono effort required to create the Challenge, and the vendor disinclination to participate, I don’t expect there will be a Detection Challenge at S4x20. The only thing that would change that is a proactive action by enough vendors committing to compete if a Challenge is created. Short of the vendors saying we want to a Challenge, it is unfortunately done.
We are toying with ways to integrate some of the Challenge aspects into the S4x20 CTF, and we are always cooking up new ideas to try out at S4.
See My Past Articles, Podcasts and Videos on the ICS Detection market