Thomas H. Ptacek of Matasano Security asked a question to all of the Twits following his tqbf twitter account:
Quick, Twitterverse. You put a “canary” email account in your database to see if you get owned. You start getting spam to it. NOW WHAT?
I only had time for a quick exchange and I couldn’t follow along with all the responses in real time. My answer basically boiled down to:
@tqbf I’m coming in late but here is my quick answer Don’t panic if you have implemented a canary you have a IR plan. Follow it.
Let’s set up the situation a little bit to ensure that everybody understands what Thomas was referring to as a “canary.” I am fairly certain that most everybody understands that miners use to take a canary with them to act as an early warning system in the event of toxic gases. In this case an extra entry has been created in a database that is supplying information to another application (internal or external) to help provide a warning that the application or database has been compromised. The thought process being that the only way that knowledge of the email address associated with the “canary” entry would be if there was a compromise. Although you can tell from his tone (especially if you understand Thomas’ usual approach to showing that something does not make much sense), Ryan Russell pointed out that Thomas was most likely making a statement that the canary was “stupid cause he thinks there’s no IR that makes sense or that anyone would do.”
I am not going to recap any of the responses to and from Thomas as you can do it yourself should you have time to click back and forth through the twitter accounts Thomas responded to during the exchange. I would like to follow up on my comment and how I think a canary record in a database may be helpful.
What does the canary in this situation really represent? Basically, it is a mechanism to help you identify when your other controls have failed for any number of reasons:
- poor configuration
- bypass
- not monitoring the attack vector
- user error
- malicious insider
- [insert your own here]
It is one more piece of the defense in depth that may come in handy. It is not designed to provide any more information other than the fact that an event has occurred that has exposed the canary to the real world. Utilizing it an a number of ways may help provide more information about when, how, and what happened but most likely if will not.
What should you do if you start receiving email to a canary email address? I stand by my very first statement. DO NOT PANIC. You received an email to an account. That is all. Until more information about the activity can be determined there is no specific methodology, outside of an incident response plan, that should be implemented. The database server and application server do not need to be imaged and forensically examined on this information alone. You could do so, but it is a knee-jerk reaction that could be, and more than likely is, wasting valuable time.
Hopefully, if your organization has implemented a canary it has also developed and tested an incident response plan. The person monitoring the email account who has detected the anomaly should initiate this plan. More than likely this will initially pull a limited number of key personnel together to evaluation the situation, identify what is happening, and decide on a course of action. If your company does not have an incident response plan then you should read this paragraph again and use it as the basis for your initial reaction. Pulling the proper personnel together right off the bat will help narrow down the events that have transpired and thereby initiating a more focused and proactive response. More than likely that will begin with more information gathering to get a better understanding of the situation.
Now some people might still argue that to be on the safe side the database and web applications systems should be forensically imaged or even taken off-line. I noticed from some of Thomas’ tweets while getting the links for this post that he did receive some of these responses and he pointed out, rightfully so, that it more than likely would not be cost effect, even extremely detrimental, to take either system offline. Although a system can be imaged while running, there is always the risk that while doing a live response and copying memory and/or all internal media that there will be a system failure thereby bringing down the database, application, or even the whole server. Now wouldn’t a incident responser look a little silly if they brought down the only web application servicing a business’ e-customers just to discover later that the email address was leaked out when an internal document detailing the canary methodology, including the email address, was leaked to the world by an internal user and their authorized/unauthorized peer-to-peer file-sharing software? That’s a BIG opps that could have potentially been avoided with a little investigation and a little less knee-jerk reaction.
After thinking on whether the canary in the database method is a technique that should be implemented I have come to the age old conclusion: “Maybe.” What it really boils down to is what other actions has an organization implemented to prepare themselves for the day that the canary account does start getting email. There will be no way to begin answering the questions pertaining to root cause for such a leak if the appropriate logging on the network, operating system, application, database, and other controls has not been implemented. If you cannot accurately determine the details behind an event then you cannot determine the details behind an incident. Having a canary in this situation would only lead to too many meetings, poor response efforts, and more money in resources and man-hours than an organization is capable of absorbing effectively. That said, with the proper controls and incident response plan in place, a canary record in a database, a canary file in a file-share, an embedded html tag in a document, or a network monitor on a dark net are a few techniques that only take a few minutes to implement but could provide valuable information about an event and possibly and incident. These could be leading contributors to detecting and reacting to an unforeseen threat or newly exposed vulnerability. Although simple, they could be the piece of the puzzle that allows an organization to close the gap between the incident and the initial response.
Go forth and do good things,
Don C. Weber
