Incident Response Lessons Learned
Following up on any project is key. Talking to all involved about what has happened, why it happened, how it could have better, what worked very well, etc is the key to improvement. So, why aren’t organizations that experience a security related incident able to prevent or curtail future incidents? Basically because they are not following up the incident with a lessons learned, or they are not doing it properly.
Incident Response should be handled like any other project. It should be managed. As most incident responders are aware, SANS GCIH course outlines 6 distinct phases of an incident response:
- Preparation
- Identification
- Containment
- Eradication
- Recovery, and
- Lessons Learned
Ah, Lessons Learned is the final step. Actually, it is circular, so the final step leads into the first step. I say leads into whereas a better metaphor might be “feeds.” Or in the terms of following flow charts, for you management types out there, the outputs of Lessons Learned are the inputs of Preparation. Here’s a question for you though, during your preparation phase did you look at the methodologies that you use to improve how you identify, prioritize, address, and follow-up on your lessons learned? Or, when you are finished with your lessons learned meeting, do you have a list of action items that have been assigned to a specific individual who understands the criteria for successfully completing the action?
I just asked two questions that took your simple lessons learned meeting from a quick five minute session to a thirty minute plus session. Although you might think that the issues will drive how long this meeting will last, in actuality that is not correct. With proper methodology, in other words a practiced plan, this meeting can still be very quick while producing the key outputs that are necessary to augment your preparation phase.
Silver Bullet time. Yes, I know, you want the silver bullet that is going to help you increase the effectiveness of your lessons learned process. Guess what!! Most likely you already have it. Do you use process management to plan your software, hardware, or infrastructure development? Bingo, then you have the means to improve your lessons learned. Start using the processes that you already have in place. I say this because it is the fastest and cheapest method to gaining control over this process.
For those of you who do not have a process in place, never fear, I have one word for you: Why. Just ask why. But ask it five times. Asking why five times is the technique for determining root cause. Gasp, Root Cause Analysis. It is used during the Six Sigma process as well as being integrated into other project development schemes. In his 5 Whys post Eric Ries explains that Taiichi Ohno of Toyota fame wrote about this technique in his book Toyota Production System: Beyond Large-Scale Production.
Now, this might seem silly to some at first. It will seem especially silly to those individuals and groups that are not use to management their projects or doing root cause analysis. This is where the longer meetings are going to come into play. Of course, this is true of any new process. New things take time to understand and get use to performing. For some people the learning curve on how to conduct themselves in these meetings is going to be a long and tough journey. But by consistently applying one of these methodologies to your lessons learned process you will find that each of your meetings is shorter and more productive. The implementors will be happier because they are being heard (if they are participating) and the managers and executives will be happier because of the increased productivity and effectiveness of the end results.
Certainly I have only touched on this topic briefly. If you have techniques that have worked to improve the effectiveness of your lessons learned meetings, please share them with us in the comments.
Go forth and do good things,
Don C. Weber
http://www.sans.org/?utm_source=web&utm_medium=text-ad&utm_content=affiliate_link1&utm_campaign=Cutaway_Security
Help support my training and travel to security conferences. Get your SANS Training and GIAC Certifications through the Security Ripcord.
You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.









February 20th, 2009 at 6:09 pm
5 Whys is really a very poor root cause analysis technique.
I’ve written about why 5-Whys doesn’t work well before.
See:
http://taproot.com/wordpress/2007/03/05/whats-wrong-with-cause-and-effect-5-whys-fault-trees/
for some ideas.
February 20th, 2009 at 6:37 pm
@Mark of taproot:
You are missing my point. I’ll start with a quote from your post “Once an investigator is trained in using TapRooT®, they find a broader range of causes…” I have no problem with people and organizations using any method to help them determine how to follow up an incident. I’m merely trying to give them a technique that they can use RIGHT NOW. Hopefully they well see your comment and review these techniques. If they like it then they can start using it. Until then, they need a stop gap solution and 5 Whys does just that.
To many organizations need to start doing lessons learned. Once they start they can start looking into methods to improve their technique. Thank you for pointing out one of those methods.
Go forth and do good things,
Don C. Weber
February 26th, 2009 at 8:27 am
Where we are at, the LL is becoming a solid fixture. It is sorting out the mundane from the sublime. Known issues, from the ITIL universe, are becoming much more clear with “workarounds” changing from duct tape and bubble gum to recommendations on controls and improved configurations. It’s a shame that remediation of known elements do not receive complete backing from management until something breaks. Granted, our LL’s are fast and from the hip, but all the stakeholders are not paying attention and showing up for our hour of education.