It is close to a universal truth that vendors in all industries do not handle their first vulnerability disclosure incident well. We now know the same is true of User Groups with the DNP3 User Group as an example.
The widespread DNP3 implementation vulnerabilities found in master and serial fuzzing by Adam Crain and Chris Sistrunk, see the S4x14 video, were initially and to a large extent treated as not a problem with DNP3. This is a great approach if you are trying to avoid any negative buzz, and the identified vulnerabilities were not related to the specification. It was when input was not validated or the protocol was violated that the DNP3 protocol stacks crashed.
However how can a user group downplay and not seriously address an issue that affects almost all of the users? Isn’t the fact that almost all of the tested DNP3 protocol stacks have implementation flaws that can lead to an attacker crashing the master from an outstation an issue that affects users? Eventually there was enough pressure for a helpful response that the DNP3 User Group promised a “set of recommendations” in a document where 80%+ said this is not a DNP3 problem. The recommendations, AN2013-004, are in a helpful and quite good 24 page document available from the DNP3 User Group.
The response to the DNP3 implementation vulns was and is determined by the DNP3 Board of Directors. The Board of Directors is largely controlled by Triangle MicroWorks (TMW), the big gorilla in DNP3. The majority of DNP3 vendor implementations use the TMW stack. The TMW outstation had some vulns, but the master actually did not. Interestingly, the testing found that most of the vendors that integrated the TMW master station protocol stack had vulns.
It’s unclear how much of the blame should be placed on the package that TMW provides their OEM customers and how much should be placed on the integration team. However it is a huge problem for TMW and its customers. They have a large deployed base of DNP3 Master Stations based on the TMW protocol stack that can be crashed from an outstation, via serial or IP comms.
Now to the politics … the DNP3 User Group elected half of their board this January. Erin Hall of TMW was up for re-election to the Board and was concerned that someone who pushed fuzz testing might get on the board. She sent a letter to the TMW customer base, here is the pull quote:
I am concerned this may be an effort to gather enough votes to move the DNP Users Group toward requiring “fuzz testing” or otherwise modify the protocol in a way that benefits fuzz testing companies, rather than benefitting the industry or the DNP3 protocol. Please read the DNP3 ICS-CERT FAQ from Triangle MicroWorks for an explanation of why I think there are better ways to improve DNP3 Security other than fuzz testing.
It could be very painful for TMW and their OEM customers, and the eventual end users, to learn how pervasive this problem is and the potential impact if an attacker can gain access to an outstation.
The DNP3 User Group board has six or seven members (this was unclear but it appears there is some emeritus position). Two members are from TMW, Erin Hall, VP of TMW and Jim Coats, President of TMW. All they need to do is swing one other Board Member, which TMW has some interesting relationships with, and TMW controls the DNP3 User Group. While it may be within the bylaws, it can’t be in the best interest of DNP3 users to have TMW dominate the future of DNP3 this way.
In fact, I would argue that the TMW has a huge conflict of interest on the fuzzing vuln issue and should have recused themselves from voting on any response to this issue.
To come full circle on this article, now that the DNP3 User Group has some experience dealing with vulnerabilities they will likely do better next time. Vendors have the same issue that the TMW members on the DNP3 Board face and eventually learn that customers expect honest and helpful information when vulnerabilities are identified.
Image by Newfrontiers