Security Ripcord

It is more than obvious now that my ShmooCon talk, Looking into the Eye of the Meter, was canceled.  Kelly Jackson Higgins in her  Dark Reading article Researchers Postpone Release Of Free Smart Meter Security Testing Tool did a good job describing what InGuardians and I can say about the topic. But even one week later there is not much news or discussion about this. Robert Former wrote an interesting blog post, Security researchers: Spawn of Satan, Necessary Evil, or Security Salvation?, about his opinion of the event. I liked it, but I am biased (Free SMACK!! Indeed.). Other than that the Smart Grid lists and media feeds have been very quiet about the whole issue. It seems to be turning into the little meter talk that faded into the night. I cannot say if this is the goal of the vendor that asked us to pull the talk, but I can say that it was my fear when they asked us to pull it from the ShmooCon venue.

I cannot provide any more details about the vendor’s issues until we speak with the vendor and get some things cleared up. I want to talk a bit about my opinions on why we decided to pull the talk at the request of a vendor. Of course, these opinions are my own and do not reflect the official opinions of InGuardians, Inc.

I wrote the Smart Meter presentation to educate. I wanted to educate security professionals that will be dealing with the fast proliferation of embedded devices. I wanted to show that following a good testing methodology and implementing the hardware analysis basics well could lead to findings that will strengthen future development and implementation of a product and associated services. ShmooCon might be considered a “hacker” conference, but the people attending this event are security professionals and researchers that will be providing guidance to information technology deployments throughout the world. By educating these people I am not teaching the public how to hack the Smart Grid, I am informing them on the basics for security assessments and providing them with the skills to pass on to their teams and organizations.

In addition to security professionals I was hoping to educate the Smart Grid-related vendors and implementors. The Smart Grid industry has come a long way in the last three years. Most of the vendors and utilities have been listening to security professionals (internal and external) and addressing issues as quickly as they can. As with all industries there are individuals and even teams of people who are afraid of information disclosure and security tools. Basically, the Smart Grid industry, although moving as quickly as they can, are still new to the positive impacts that research, education, and assessment tools can provide to their efforts. They are not familiar with Wright’s Law: “Security will not get better until tools for practical exploration of the attack surface are made available.” I believe that Richard Bejtlich put it very well in a tweet posted as I was arriving in Washington, D.C.: “Proponents argue that releasing weaponized exploits will have long term positive impact; critics worry about short/med term negative impact.” Although I was not intending on releasing a “weaponized exploit” the statement hits very close to home for all security-related research and tools.

InGuardians has experienced the effects of security-related misconceptions within the Smart Grid arena. In fact, I almost didn’t get hired at InGuardians because of the fear, uncertainty, and doubt raised by the Associated Press article: ‘Smart’ meters have security holes. The title alone was enough to raise a stir within the Smart Grid industry not to mention the fact that the reporter left out many of the positive statements made by Joshua Wright. I now have a good understanding of how Josh might have been feeling at this time. I respect him immensely for his continued courage and desire to educate.

Because of Josh’s, InGuardians’, and other experiences, I approached presenting a Smart Meter talk at ShmooCon very carefully. I first had to sell it to InGuardians. I provided our team with an outline of my presentation idea with specific emphasis on the mitigations already employed by many meter vendors and third-party solutions. Mike Poor (yes, Tom Liston, it is Mike Poor’s fault) especially liked the idea as our approach would replace Fear, Uncertainty, and Doubt with Understanding, Certainty, and Knowledge (TM, I think. ). With InGuardians’ approval I had to sell the optical toolkit (the actual basis for my presentation) and the presentation content to the Smart Grid industry. I did this by providing access to the toolkit to our contacts at as many Smart Grid vendors as I could. These people were provided access to the toolkit on January 8, 2012 (approximately, I would have to double check for accuracy). I received immediate positive feedback from the list. Robert Former of Itron provided me with an understanding of the impacts of the tool to Advanced Metering Infrastructure (AMI) solutions and helped me understand implementable mitigations. Ed Beroset of Elster provided me with code modifications to make the tool more stable and usable by meter research and development teams. Several other individual, who have asked not to be named, provided us with their thoughts on the tool as well. As we did not receive any complaints about the release of the tool we continued with developing the presentation. After intense internal and legal review, I provided a draft presentation to the same group on the Monday before the presentation. This gave the group at least five days to provide us with their input about the content. The same individuals that provided me input about the tool also provided me input about the content of the presentation. All of the input was positive and really helped tighten up the concepts and content of the presentation.

We did not hear any objections to the toolkit or presentation until the first day of ShmooCon. One of the vendors on our list contacted us with generic objections to the content and venue of the presentation. As we had almost two full days to work through their objections InGuardians began a concerted effort to identify and address their issues so that the information could be presented and the toolkit released. I am very proud of the InGuardians team and the professionalism we displayed during these discussions. There were times when I wanted to call it and move forward as planned, but cooler heads prevailed and we adhered to the request of the vendor. And looking back, keeping a cool head at these times is the key to responsible disclosure, education, and maintaining positive relationships with our vendors and clients. InGuardians strives to be an organization of thought leaders and information security professionals that can be turned to for guidance and expertise. Of course we make mistakes, but we are quick to own them and address them as swiftly and professionally as possible. With these ideals in mind we could not move forward with the tool release and presentation until all parties were comfortable. It could be said that trying to please everybody is a recipe for disaster, but we cannot ignore direct requests for additional information. Because in the long run it provides each party with a better understanding of the positive and negative impacts of publicly releasing information and toolkits relating to specific products or solutions.

The result of all of this is that my Smart Meter presentation and the release of the Smart Meter Assessment Communication Kit (SMACK) has been delayed until further notice. I have already received an offer to give the presentation at the next Smart Grid Security Summit in Atlanta. I am hoping we can work through the vendor issues before they have to fill the slot with another presentation. I am also hoping to present this at another information security-related conference such as Black Hat USA and/or DefCon so that this information can educate the future security professionals that will be moving into Smart Grid-related positions or conducting authorized research on Smart Grid solutions. Of course this requires that the presentation is accepted for these venues.

I am also hoping the the vendor will come forward with a statement that they requested us to hold the presentation and release of the toolkit and InGuardians worked with them in good faith. This would show the public that they have a concern about the information being provided and that all parties are working to address the issues and concerns before moving forward. This would have a positive impact on the Smart Grid industry and security researchers.It would demonstrate that we are working as a team and not playing a fancy political game to suppress information or teach criminals how to hack the Smart Grid.

Go forth and do good things,

Don C. Weber

All of the opinions expressed in this blog post are mine alone. They do not represent the opinions of my employer, InGuardians, Inc.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.