The Sessions
Digital Bond Labs appeared at Black Hat Sessions in Ede, Netherlands. We gave a talk on vulnerability inheritance in PLCs, and also discussed some of the challenges associated with removing vulnerable internet-connected control systems from their wide attack surface.
The conference was a well-run one-day event put on by Madison Gurkha. Ede is a fairly small town, but thanks to being in the Netherlands is easily reached by train (or bicycle). BHS has been increasing in size steadily over the years, and this year’s attendance was just shy of 400 total conference-goers. While the keynote talk was in Dutch and thus incomprehensible to me, there were three good technical talks in English, including a talk by ERPScan. S4 alumni may remember ERPScan as the employer of Alexander Bolshev when he gave his excellent HART security research talk in 2014.
ERPScan’s main focus is on SAP systems. A little LinkedIn searching shows some large manufacturers in the United States that are using SAP (although, it is still a more common business app ecosystem in Europe). From the exploit demonstrations at BHS, it also has fairly terrible security practices and likely little internal negative testing. Dmitry Chastuhin’s SAP talk detailed some of the programming bugs, many of which were disclosed and fixed in 2014 and 2015. One of the bugs in particular — an administrative access bypass using an HTTP HEAD request, should give users of the system a bit of pause.
The ‘hallwaycon’ at BHS was also exciting. There I met the creators of the Secure Software Foundation, which helps vendors implement the SDLC, chatted with testers at Underwriter Laboratories (which just this week announced that they will begin certifying devices for security), and even met utility employees who were negatively testing their new equipment as part of their acquisition process. (This last bit was probably my happiest conversation in years.)
There was even a PGP key signing party at BHS. These are a common event at cons in Europe, and something that we should do more of in the USA. So, look for a key signing party at S4x16…
Labs’ Talk
For our talk, we did some extra research on internet-connected PLCs in the Netherlands. Labs identified 50 CoDeSys PLCs that were vulnerable to CVE-2012-6068 and CVE-2012-6069 in Holland. We found the owners of a few of the controllers, and reached out to NCSC as well as Dutch ISP incident response teams prior to the conference to help find and contact the others.
One of the vulnerable controllers was removed prior to the talk, which is good, as it was the most interesting — a part of Holland’s public transportation infrastructure. The others remain online, but also appear to be a lot less important to public safety.
The most interesting part of ‘scouring the Internet’ in my opinion comes from a paper that Eireann Leverett and I produced in 2013: a metric detailing the cost of finding vulnerable systems (this idea was Eireann’s). When we first searched the Internet for CoDeSys PLCs, it cost us over $1.00USD per vulnerable device found. This cost was primarily born by expenses for servers situated in a few points around the world, and the time it took to prune and verify results. Since we shared our script with John Matherly of Shodan, the cost has come down to $0.01USD per vulnerable device found. Not only are there more vulnerable CoDeSys controllers on the Internet today than in 2013, but the cost to find them is a trivial $20USD paid account on Shodan, followed by an equally trivial search query.
An up and coming area here is ‘whose responsibility is it?’ when it comes to attaching insecure-by-design control systems to the Internet. It’s not really fair to blame the victim (although, end users really *should* know better).
So far, there is little expectation that ISPs or even governments will step in to help. Certainly there are government regulations in many countries regarding securing ‘critical infrastructure,’ although if history is any lesson there will be safety and life-critical doodads left exposed due to poor classification or end users simply not following the regulations. In the United States for example we have found feeder management relays and even substation bay controllers (for very small substations) that are directly connected to the Internet. Neither of these classify as critical, but would cause a bit of panic if not secured.
ISPs for their part have very little interest in sharing customer contact information with their government, let alone private researchers, unless they are legally compelled to do so. ISPs also have little desire to search their own networks for vulnerable devices, as it may upset their customers to know that their ISP is sending unsolicited traffic to their systems.
I think it is fair to say that eventually one of these things will change. That is, governments will pass a law mandating that ISPs share ownership information for vulnerable systems, or ISPs will start performing security scanning and protection (perhaps via a legal compulsion). The current system of security researchers volunteering our efforts to secure infrastructure isn’t sustainable. There is no business case for us to do it (although we feel it is the ‘right thing’ to do), and it is rarely very effective for us. Where the cost metric becomes important is precisely in a rationalization for a legal mandate: the cost to search for vulnerable systems is essentially $0 now, which removes the financial barrier to such a program.
In all, Black Hat Sessions was a great event, even for a non-Dutch speaker. It’s definitely a must-go conference for anyone in the Netherlands.
image by archetypefotografie