Security Ripcord

When I started working for IBM’s Emergency Response Team I was a little intimidated about walking into a client’s environment and quickly providing incident response leadership. Luckily I was trained by Chris Pogue and Harlan Carvey to consider three things when I got on-site:

• What are you trying to answer?
• What data do you need to answer it?
• How do you get that data?

Obviously the first question is the hardest to answer (although, perhaps, not the most challenging). I often find myself having Car Talk-style conversations where you talk about the situation for an hour or two and then somebody says something to the effect of: “Oh, yeah. And we have this service running over here.”

It always amazes me how similar many of the security issues are across companies with decent security programs and teams. The security teams work hard to identify situations before they happen but incidents still occur and the teams struggle with pulling all of the information together. Disciplined incident response comes with experience. Not just experience within the security team, but experience throughout the organization. Most companies have information technology (IT) divisions with silo-ed experience (I am not saying it is bad, just natural). The web application developers and administrators know the web applications and web logs. Database administrators know the database logs, how verbose they are, how to retrieve them, and what can be turned on and off. Network administrators know what is logging where, what has been disabled, what can be turned up. Etc, Etc. Of course there is some overlap, but this example demonstrates that a lot of internal interaction may be required to initiate an incident response.

This silo-ing can complicate things but it is often manageable. The situations that prove to be more challenging are those that involve external services and equipment. Security teams are not usually consulted when service level agreements with external entities are formulated. Any third-party organization worth their salt will be prepared to provide information and action during critical times. But experience is the only way to determine the proper channels and methods of request to initiate the actions necessary to facilitate an efficient incident response. Acquiring data, removing services, increasing logging, and physically accessing a facility are just a few of the things that could increase the gap between incident identification and response.

Another thing that comes into play during an active incident response is the review of unique data types. I have assisted in several instances where an incident involved a web application running on an Apache server. The Apache web logs did not include any POST information which may have provided artifacts relating to the incident. As these web logs were the primary method for detecting anomalies the security team quickly realized that with the default configuration they may have been missing a few things. Digging a little deeper by including web application administrators and developers additional logs were identified. In a recent case the development team used Apache log4j to produce excellent activity information for debugging. As it did not introduce a performance hit to the server the developers just left the debugging code alone. The end result was a detailed log of the web server activity. The only real hick-up was that each entry was undocumented and multi-lined making it difficult to review by common automated processes. But, in the end, very helpful.

So, again, we come back around to the same incident response-related conclusion: Preparation is KEY. Understanding how things interact and who is involved is critical to reducing the time involved during each step of an incident response. But where to start with preparation? The results of an actual incident or a penetration test provide excellent examples and supporting data. Incident response scenarios are also very helpful but can be difficult to design and get everybody involved. To be honest, I have been a big advocate of “incident response scenarios” and I have told many people to develop them. But, looking back, I realize how hard it is for an organization to do this. Heck, it was hard for me when I was specifically thinking about and devoting time to developing a good incident response scenario. Therefore I am going to change my future recommendations from  developing incident response scenarios to performing system walk-throughs.

I made this decision when I found myself leaning back in a chair, looking at a white-boarded diagram of a system (by this I mean a group of resources combined for a specific purpose), and trying to determine what we needed to understand what might have happened during an incident. It was excellent because I also found myself and the security team asking questions to which we did not have immediate answers. I also found myself thinking about how I would have accomplished a specific set of actions relating to the incident. Coming up with these possibilities lead me to asking additional questions and identifying other resources-of-concern and data that could be used to identify useful network and system artifacts.

Therefore, my recommendation is that security teams periodically perform a system walk-through for various IT implementations in their organization. See how systems walk-throughs work and let us know your experiences. Pick one thing at a time when you start and see where it takes you. Focus on the things that will provide you information about a specific activity. Brain-storm situations and talk about the system and network-based artifacts that would be helpful. Folllow up on these ideas and pull in personnel to provide additional information when necessary. This will increase their experience and begin the collaboration process that may prove vitally important in future incidents. I also believe that these interactions will help identify risks, weaknesses, and vulnerabilities that can be easily addressed and increase the overall security of the organization.

Go forth and do good things,

Don C. Weber

Mentoring can be a powerful learning tool for learning specific topics. I have been thinking about mentoring a little bit because I have often found myself thinking that a mentor would be beneficial to my technological and managerial growth.  From my experiences I have determined there are a few requirements to setting up a good mentoring situation. Now, I am not a professional mentor, so, your mileage may vary.

• Focus: a broad range of mentoring subjects do not really work. A mentoring experience should be focused on a specific area.
• Time: both mentor and mentee must have time and inclination for the mentoring experience. Mentoring requires consistent meetings and reviews.
• Experience: A mentor must have knowledge of the subject and mentoring. A mentee must be able to learn the subject and be willing to branch out without encouragement.
• Desire: Both parties need to have a strong desire to participate from the beginning to the end of the mentoring experience.
• End State: both mentor and mentee must have an understanding of when the mentoring experience will end.  This may be after a specific set of goals is reached or a after a specific time period.

Personally, I don’t think that a blossoming Information Technology (IT) Security professional needs mentoring outside of the educational practices implemented by their place of employment. But that is the key: place of employment. Without a job it is difficult to implement the requirements that I listed above. Obviously this means that my list should include the additional requirement of: a job. Of course, a job is generally why a blossoming IT Security professional wants to enter into a mentoring experience.

How do people attempting to enter the IT Security profession get past this gap? My thoughts: effort, tenacity, and the will to teach. A well rounded IT Security professional starts by knowing a lot of things. They slowly, via experience, move to knowing a lot about a lot of things and eventually realize that they do not know and have not done everything. Once they realize they do not know everything they accept the fact that they will be learning during their whole career. I also believe that continuous effort and vocalism are requirements for becoming a well-rounded IT Security professional. Not only does this get you noticed by your peers, it expresses confidence to other administrators and managers.

As IT Security professionals mature they tend to filter into specific areas of focus. This usually means that the influential people in their lives become more narrow and they start to enter into good peer-mentoring relationships. By “peer-mentoring” I mean that each person grows from the other’s experiences. They drive each other to dig deeper and accomplish more. Eventually, people start developing other interests and the “peer-mentoring” relationships shift. But the excellent thing is that, generally, the previous relationships (business or friend) remain strong and continue throughout the years. At some point, interests will diverge again and collaboration will continue.

So, if you are just getting into IT Security or attempting to transition to a new focus area, do not get stifled or discouraged by the lack of mentoring opportunities. Rather, put yourself out there. Start working on personal projects. Tell people about your experiences, failures and success (because BOTH are important to the learning process). Do not be discouraged by critical feed back and have fun even if the subject is hard from start to finish. Remember, experience comes with time and effort. Learning the hard way helps people understand the easy and efficient way. In the end, people will notice and you will have achieved your goal.

Go forth and do good things,

Don C. Weber

Slow week, again.  My wife is loving it that I have been at home recently.  It’s good to be wanted.

Leadership

I have been plowing through the Enderverse over the past month.  My wife finds it weird how I get fixated on reading.  I am not surprised that it is new to her because I have not really read a non-technical book since completing my Master Degree in 2005.  The whole time she has known me the only thing she has really seen me get fixated on (other than task completion) is moving through a new video game.  I guess she should have seen the signs.

Reading the full Ender and Bean series has helped me understand Ender’s Game a lot better.  These books show the importance of understanding yourself, understanding the people you work with, understanding the people you work against, strategy, and many other leadership qualities.  Certainly you can get these from other books, but the Enderverse does a good job of showing enough of both sides of the story to understand motivation.  Which I am finding to be the biggesst take-away from this series.

Orson Scott Card demonstrates how effective it is when leaders understand an individual’s motivation.  Actually, it goes a little deeper that than.  He has his main and most successful characters leading by understanding what the people around them actually “need.”  They take those needs into consideration for their own plans and actions.  Whether it is a personal relationship or it is military strategy.  These characters are most successful when they step back, take a moment for consideration, and modify their plans and judgments (when necessary) to accommodate the needs of the people around them.  Doing so usually wins that character the professional respect, friendship, and sometimes devotion.

How can we use this in our professional and personal lives?  Well, I think the best away to do this is to take a few seconds before answering questions or providing opinions during conversations (live or digital).  A few extra seconds pause during a phone call is not as long as it actually seems on your end.  Pauses are natural in face-to-face conversations because it is easy to see when somebody is thinking.  If people on the other end of the line do get impatient, just ask for a second while you are thinking or formulating a response.  It will be a rare occasion when somebody gets offended and it will be more likely that they respect you for being considerate and thoughtful.  Email and IM is another great example for a pause.  Get up, walk around, and don’t answer until you have given a little thought to your position in combination with the other person’s needs.    You can even do these types of things in meetings and one-on-one conversations.  Taking a few moments to lean back and ingest the facts of the conversation is easy and common.  For heated conversations or times when you need more time to think about it, use the bio-break method to get some extra time.  Get up, go to the bathroom, get a glass of water, and think about your position, your stratagy, and weight that with the needs of others involved in the situation.  It doesn’t matter if others know you are just using it as an excuse, nobody is going to deny anybody a bio-break.

Quick Tip

I have been working on a bit of code for several months now.  I am getting into the fine-tuned debugging and I hit a snag.  One piece of advice that the mentor for my project gave me was “never be afraid to revisit and rewrite your code from scratch.”  Well, I am not there yet.  But I am to the point where I understand that I may have made a mistake with my implementation and I need to determine how much I need to change.  Of course, now that the code is at 1855 lines (with comments) it is a little harder to scroll back and forth to find all of the areas that need to be adjusted for the implementation change.

I have decided to fall back to printing and editing by pencil.  It should not be too hard but 1855 lines of code divided by 80 lines per page is 23 pages of code.  Ouch.  Of course there are easier ways to do this.  People have been writing and printing code for years and there are all types of tips and tricks.  *nix OSes are naturally full of different methods for converting text to printer and other document formats.  (Now that I think about it that sounds like the potential for vulnerable programs and scripts is huge….hmmm…but I digress).  Actually I knew this already and I use to have a nice little script that would print two pages per sheet, in landscape, and modify the font so that it was small but not too small to read and edit.  A quick look showed me that I had disposed of that code a long time ago (several OSes ago).  But, the Internet is rich with ideas and methods.  I ran through some instances of enscript (not to be confused with EnCase’s scripting language) which seems to be the defacto method nowadays.  But for some reason it just wasn’t working for me.  AND it was not the method I used before (we are all creatures of habit) so I continued my search until I found my one true-code-printing-love: a2ps. A quick review of the very helpful man page gave me the syntax I needed to convert the file I want to print to a postscript file  (my computer is not connected to a printer right now and I have to use a friend’s).  I ended up with the following command line (I will let you review the man page to understand the options):

a2ps -2rjC -Epython -o ptr-file.ps ptr-file.py

Easy, but what if my friend’s system cannot read Postscript files?  Doubtful but just to be certain I decided to quickly convert to PDF via command line.

ps2pdf ptr-file.ps ptr-file.pdf

Yes, yes.  It would have been much easier to combine these two commands.

a2ps -2rjC -Epython ptr-file.py | ps2pdf – ptr-file.pdf

Enjoy.

Personal Input

Read more non-technical books.  It is good for you and it is good for your kids (if you have them) to see you reading instead of watching TV, or cleaning the house, or doing yard work, or playing with them, or……dang it!!!! 

Go forth and do good things,

Don C. Weber

Slow week, lots of programming.  Gonna rain this weekend.  Looks like we’ll be hanging out at the library and aquarium.

Leadership

During my bike ride on Monday I heard a blast from the past.

“Raise Your Voice” – Bad Religion

fa fa fafa fa fa fafa Raise Your Voice!
Don’t be played like someone else’s board game
Don’t be classed out like some desolate redoubt
Don’t be misled you’ve got alot on your head
And nobody’s gonna pay attention when you are dead
So: fa fa fafa fa fa fafa Raise Your Voice!
It’s the primary rule, you gotta wanna be fooled
It’s our daunted restraint that keeps us silent in shame
It’s our nature to be adversarial and free
Our evolution didn’t hinge on passivity
fa fa fafa fa fa fafa Raise Your Voice!

Actually, the song from the album begins with the lead singer stating “I think this is song we could redo in every language of every country we go to.”  True.  I think that every security professional can learn a little from these lyrics as well.

What role do security professionals play?  Auditor, monitors, engineers?  Actually, I think we are just members of the team fulfilling specific roles to keep our organization operating smoothly.  But we cannot do so in silence.  We cannot just push out reports, scan  outputs, and presentations on how people and the organization should operate.  These things are too easy to dismiss.  No, we have to do the thing that some of us dread and many more people don’t want to happen.  We need to “raise our voice.”  We need to talk about the issues.  Have open discussions that challenge the status-quo.  We need to confront common issues with new ideas and methods.

What do we accomplish when we are doing this?  Are we trying to get our way no matter what the cost?  No.  We are trying to open minds and continuously adapt and evolve.  We are verbally raising issues so that the conversations lift from the paper and actively carry on in other discussions.  We are forcing people to actively observe the current situation, address our recommendations, accept or counter these recommendations, and improve themselves and the organization.

One trap that is easy to fall into, however, is thinking that we are always in the right.  That our ideas are the correct direction.  I challenge you to “raise your voice” while also keeping your mind open to new information and other possibilities.  I challenge you to find a way to persuade those that don’t understand while also realizing that you may be one of those who does not have a clear vision of the full picture.  I challenge you to be confident yet humble in your expertise and your efforts to improve your organization and yourself.

I challenge you to “raise your voice,” no matter who is in the room.

Training

No training developed this week.  I did, however, take a few minutes to take last weeks quick tip on HBGary’s Fast Dump Pro and turned it into a training document for the rest of the analysts in my team.  Although the tool is pretty easy to use having an internal document for some of the tools that you don’t use on a regular basis is very helpful.  This allows you to quickly re-familiarize yourself with the tool.  It also helps train new personnel to the methodologies of the team.

How can you contribute to your team?  Isn’t there something that you could quickly write up that would benefit everybody?  If you don’t have that type of system in your team, would creating one help?

“Raise your voice.”

Quick Tip

You will need Mark Hammond’s pywin32 and Tim Golden’s wmi.py for this tip.  Basically, I want to show you an easy way to get exactly the system information that you want.  It will be easy to get all of the system information using Win32_OperatingSystem.  Using Tim’s method you can directly access the specific fields that you want. (inserted periods to help with proper indention due to WordPress stripping whitespace and my need for Python-structure OCD)

import wmi
c = wmi.WMI()
for os in c.Win32_OperatingSystem():
…print os.Caption

But this means that you either have to request each on individually.  This is not so bad and easy if you just want a few specific items.  But what if you want more than a few bits of information?  And what if Microsoft changes the information provided across systems?  You want to be sure that you can access that information without the script failing on you.

To handle this I suggest using a tuple to hold the values of the fields used by Win32_OperatingSystem.  Then run through the tuple using hasattr and getattr to pull the information provided by your call to WMI for the Win32_OperatingSystem information.  Here is an example script .  Of course you don’t have to include everything (as I have done here for clarity).  You can select the fields that are most important to you.  Or, you can include them all and comment out the ones you don’t want.  That way they are easily added in the future when you discover a need.

import wmi

Win32_OperatingSystem_Fields = (
‘BootDevice’,
‘BuildNumber’,
‘BuildType’,
‘Caption’,
‘CodeSet’,
‘CountryCode’,
‘CreationClassName’,
‘CSCreationClassName’,
‘CSDVersion’,
‘CSName’,
‘CurrentTimeZone’,
‘DataExecutionPrevention_Available’,
‘DataExecutionPrevention_32BitApplications’,
‘DataExecutionPrevention_Drivers’,
‘DataExecutionPrevention_SupportPolicy’,
‘Debug’,
‘Description’,
‘Distributed’,
‘EncryptionLevel;’,
‘ForegroundApplicationBoost’,
‘FreePhysicalMemory’,
‘FreeSpaceInPagingFiles’,
‘FreeVirtualMemory’,
‘InstallDate’,
‘LargeSystemCache’,
‘LastBootUpTime’,
‘LocalDateTime’,
‘Locale’,
‘Manufacturer’,
‘MaxNumberOfProcesses’,
‘MaxProcessMemorySize’,
‘MUILanguages’,
‘Name’,
‘NumberOfLicensedUsers’,
‘NumberOfProcesses’,
‘NumberOfUsers’,
‘OperatingSystemSKU’,
‘Organization’,
‘OSArchitecture’,
‘OSLanguage’,
‘OSProductSuite’,
‘OSType’,
‘OtherTypeDescription’,
‘PAEEnabled’,
‘PlusProductID’,
‘PlusVersionNumber’,
‘Primary’,
‘ProductType’,
‘RegisteredUser’,
‘SerialNumber’,
‘ServicePackMajorVersion’,
‘ServicePackMinorVersion’,
‘SizeStoredInPagingFiles’,
‘Status’,
‘SuiteMask’,
‘SystemDevice’,
‘SystemDirectory’,
‘SystemDrive’,
‘TotalSwapSpaceSize’,
‘TotalVirtualMemorySize’,
‘TotalVisibleMemorySize’,
‘Version’,
‘WindowsDirectory’
)

class sysWMI():
…def __init__(self):
……self.wmiObj = wmi.WMI()

…def getSysInfo(self):
……info = {}
……for obj in range(len(Win32_OperatingSystem_Fields)):
………if hasattr(inf, Win32_OperatingSystem_Fields[obj]):
…………print Win32_OperatingSystem_Fields[obj] + “: ” + str(getattr(inf, Win32_OperatingSystem_Fields[obj]))

sysInfo = sysWMI()
sysInfo.getSysInfo()

Personal Input

Happy Birthday #4, Collier!!!   I love you, son.  You are a great son and I am proud to be your father. Although sometimes I raise my voice TO you, I raise my voice FOR you everyday.

Go forth and do good things,

Don C. Weber

Andy Willingham, or Andy ITGuy as some of you know him, posted an interesting question in the Security Catalyst forums the other day in a post titled “Is it really ‘Game Over’?”

I attended GFIRST this week and many of the presentations gave the impression that we have hit to point where it’s “game over”. None of them ever said that exactly but Amit Yoran of NetWitness came pretty close. He said that we have already lost and it’s going to get worse before it gets better.

What are you seeing in your “security life”?
Is it that bad? Is it almost that bad?
What do you think we need to do to change the course of things?

… (Go see his opinion in the post)

Now, I haven’t been contributing to the Security Catalyst forums recently (although I know I should be), but I couldn’t let this one slip by without a comment.  Basically I take the stand point that there is no “Game Over” or even “Win or Lose” in the information technology security industry.  There is only “Adapt and Then Evolve.”   Some people might consider this “Playing Catch-up” and it can seem like it if you want to take a pessimistic attitude towards it (we all do multiple times during our careers).  I, however, view it as a constant struggle like life.  We aren’t here to “Win or Lose.”  There is not an “End Game” scenario.  There is only the constant struggle to survive the best way that we can and, while we are at it, make sure that good people are not preyed on by the scumbags in every society.  If we are very lucky, we will also make enough to support our families comfortably while they are dealing with their life struggles.

I have provided a few more details in my reply to Andy’s forum post.  Go, read it, chew on it a bit, then provide your input so that others can learn and grow.  After all, by doing so you are helping others “Adapting and Evolving” by being a Security Catalyst.

Go forth and do good things,

Don C. Weber

Okay, I have been out of the blogging game a little too long.  It is time for me to start generating some more content.  So we will begin by doing a weekly wrap-up.  Basically I am going to go over a few of the things that occurred during my week.  I might even just set up a draft and populate as I go the hit submit on Friday afternoon.  As everybody will have started drinking by then my post should be well received.

I’m going to start out a little formless.  Hopefully this will start taking a  little better shape as we move forward.

Leadership

Lead from the front.  Some people might not like it, but the only reason they have time to bitch about it is because they are not doing it themselves.  You only learn from your mistakes.  If you aren’t leading you aren’t making enough mistakes to challenge yourself and provide yourself with enough opportunities to improve.

Training

I finally got around to using Wink the other day.  I developed some training using the same old Power Point slides everybody seems to generate.  You know the type, all words and no joy….er…pictures.  Even when you have a screen shot in PPTs it is just more words.  So, at the request of one of my team, I branched out with some flash video.  Of course Irongeek has been doing this for years and provides a walk through of what and how he does it: How I make Hacking Illustrated Videos.  Luckily I didn’t have to refer to Irongeek’s site as Wink was very easy to use.  I generated a quick training on doing a network capture using tcpdump.  Unfortunately I did it on my work system so I cannot provide it here.  But the point I wanted to make was that it was very easy.  Secondly, after pulling the video off my Linux system and onto my Windows system for emailing to the team, I noticed that I could import the Wink project back into the Windows version and add audio.  Now, I haven’t tried this completely, but it seemed straight forward.  So, hopefully, we will be seeing some of this later.  Next time I’ll generate the video on my own box and I’ll be able to provide it here.  Besides, everybody already knows how to do network capture with tcpdump, right?

Quick Tip

For the training I was just talking about I had to create a 10MB file.  It didn’t require any real data, I just needed some random bits but in a large file for demonstrating the file splitting functionality of tcpdump.  So I tried using “dd” to fill up the 10MB file using the following command.

# dd if=/dev/random 0f=./test_10MB.dat

I was doing this on a Ubuntu Linux box.  Although this seems like it would work, and there are plenty of search results that say to do it using this technique, this does not work on Ubuntu Linux.  All this does is create a 4KB file.  I tried all kinds of different concatenation techniques.  But I couldn’t get a 10MB file.

It turns out that Linux systems do not produce enough entropy to fill a 10MB file quickly.  If the Wikipedia /dev/random entry is to be trusted we see that “When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered. (Source: Linux Programmer’s Manual, section 4)”.

So, left out to dry I queried a friend of mine (Thank you, Schism) and he pointed out that the best way to do this is to use the following command.

# dd if=/dev/urandom 0f=./test_10MB.dat

Which, of course, works like a charm.  He was also quick to point out that this does not provide the “randomness” as is produced by /dev/random, but for my purposes it worked well enough.

Development

I have been working with Python programming a bit over the last few months.  It has been interesting watching the program I have been working transform into something useful.  I wish that I could provide a little more information about it, but it is still an ongoing project, so I’ll refrain.

One thing I can talk about is building function calls in Python.  I am not sure if I am using the right terminology for it, but basically what I mean is creating a list of functions and then calling them based on the results from some other function.  For instance if you can define a table or list of functions like this:

func_table = ( one, two, three )

Then create a function of those names (necessary indent removed by WordPress, so just pretend)

def one():

print “one”

def two():

print “two”

def three():

print “three”

Finally you can call each function by referencing them via the list.

for i in range (3):

func_table[i]()

When run this will produce

one

two

three

Special note: if you are returning information from these functions then you will need to store them in a variable before using or returning.  At least that is my experience.

Not very exciting until you start thinking about using tests instead of iterating through the loop.  This can be used to clean up complex code very nicely.  My understanding is that this will also help with optimizing the execution of the code as well.  What the magic balance between this method and several “if” statements is, I do not know.

Can anybody describe to me a good method of testing this optimization?  I don’t know enough about programming to come up with a complex enough task to challenge the CPU and memory of my system and produce viable results.  If you have a recommendation leave a comment and I will test and post results.  Or, you can run it yourself and post your results in the comments.  Either way it will be beneficial to us all.

BTW, thank you, Invisigoth.  I learned this from reviewing your code.

Personal Input

Not much today as I need to get down stairs with the boys.

Go forth and do good things,

Don C. Weber

Following up on any project is key.  Talking to all involved about what has happened, why it happened, how it could have better, what worked very well, etc is the key to improvement.  So, why aren’t organizations that experience a security related incident able to prevent or curtail future incidents?  Basically because they are not following up the incident with a lessons learned, or they are not doing it properly.

Incident Response should be handled like any other project.  It should be managed.  As most incident responders are aware, SANS GCIH course outlines 6 distinct phases of an incident response:

1. Preparation
2. Identification
3. Containment
4. Eradication
5. Recovery, and
6. Lessons Learned

Ah, Lessons Learned is the final step.  Actually, it is circular, so the final step leads into the first step.   I say leads into whereas a better metaphor might be “feeds.”  Or in the terms of following flow charts, for you management types out there, the outputs of Lessons Learned are the inputs of Preparation.  Here’s a question for you though, during your preparation phase did you look at the methodologies that you use to improve how you identify, prioritize, address, and follow-up on your lessons learned?  Or, when you are finished with your lessons learned meeting, do you have a list of action items that have been assigned to a specific individual who understands the criteria for successfully completing the action?

I just asked two questions that took your simple lessons learned meeting from a quick five minute session to a thirty minute plus session.  Although you might think that the issues will drive how long this meeting will last, in actuality that is not correct.  With proper methodology, in other words a practiced plan, this meeting can still be very quick while producing the key outputs that are necessary to augment your preparation phase.

Silver Bullet time.  Yes, I know, you want the silver bullet that is going to help you increase the effectiveness of your lessons learned process.  Guess what!!  Most likely you already have it.  Do you use process management to plan your software, hardware, or infrastructure development?  Bingo, then you have the means to improve your lessons learned.  Start using the processes that you already have in place.  I say this because it is the fastest and cheapest method to gaining control over this process.

For those of you who do not have a process in place, never fear, I have one word for you: Why.  Just ask why.  But ask it five times.  Asking why five times is the technique for determining root cause.  Gasp, Root Cause Analysis.  It is used during the Six Sigma process as well as being integrated into other project development schemes.  In his 5 Whys post Eric Ries explains that Taiichi Ohno of Toyota fame wrote about this technique in his book Toyota Production System: Beyond Large-Scale Production.

Now, this might seem silly to some at first.  It will seem especially silly to those individuals and groups that are not use to management their projects or doing root cause analysis.  This is where the longer meetings are going to come into play.  Of course, this is true of any new process.  New things take time to understand and get use to performing.  For some people the learning curve on how to conduct themselves in these meetings is going to be a long and tough journey.  But by consistently applying one of these methodologies to your lessons learned process you will find that each of your meetings is shorter and more productive.  The implementors will be happier because they are being heard (if they are participating) and the managers and executives will be happier because of the increased productivity and effectiveness of the end results.

Certainly I have only touched on this topic briefly.  If you have techniques that have worked to improve the effectiveness of your lessons learned meetings, please share them with us in the comments.

Go forth and do good things,

Don C. Weber

http://www.sans.org/?utm_source=web&utm_medium=text-ad&utm_content=affiliate_link1&utm_campaign=Cutaway_Security

Not all incident responses go according to plan.  This means that a responder has to have multiple methods for accomplishing tasks.  Harlan Carvey brings this up on his blog often and it is why Ed Skoudis created his famous WMIC Kung Fu write-up and turned it into a SANS Course: Windows Command-Line Kung Fu In-Depth for Info Sec Pros.

On a recent engagement this fact came into play.  I went on site knowing that I would have multiple types of Unix and Linux operating systems to potentially acquire.  From the customer’s description I also knew that I would be dealing with some older equipment but nothing from their descriptions seemed out of the ordinary.  Once I  got on site I reviewed the situation with the customer and everything seem normal.  As I was not worried about collecting volatile data from the system I requested the customer set up a maintenance window and temporarily take the systems off line in a manner that provided the least amount of impact to their operations.

All went well until the system administrator handed me the hard drive.  It all seemed fine at first until I tried to plug the drive into the adapter that was connected to my write-blocker.  The pins didn’t match up.  Although there were plenty of pins across, the layers were placed too close together.  It was fine for the blade server that it came out of, but none of my adapters could plug into this configuration.  At first I figured this was no big deal.  I decided to boot to a Linux boot disk and just copy the drive from there to a USB hard drive.  Well, there was two problems with that: 1. The blade server did not have a CD-ROM drive, and 2. the USB port was version 1.1 and would not recognize the external hard drive that I had purchased (no big deal because 12MB per second would not cut it either!!).  So, plan one == bust and plan two == bust.

So what was plan three?  Well, dump over NFS using native commands, of course.  Luckily the system administrator was well versed in NFS and the commands we were about to use.  So there was not a lot of explaining, I didn’t have to type commands, and my only real job (as it should be in this type of situation) was mindful and guided keystroke monitoring.  Here is what we did.

• To start off, please review this great guide on setting up NFS via the Ubuntu community wiki.  It does not dive deeply into NFS.  It just gives the basics on how to get it up and running on a live system or Live CD.  Of course, mileage may vary depending on distro, so beware.  Once it was all set up we navigated to the correct directories and prepared to acquire.  When all was done the system we were acquiring had a remote share mounted to /tmp/target.
• Before we started copying files we needed to keep track of everything we did.  Of course we could have done that as soon as we logged into the system, but as it was the system administrator performing the actions and this was not necessarily a “Live Acquisition” it was not completely necessary and we needed to have a good place, like an NFS mounted directory, to copy our files.  So we changed to the /tmp/target directory and ran the script command to record our actions and sent it to an appropriately titled file.
• Next we needed to record the system time.  Although we did note the BIOS time when the system was booting up, we need to record the time displayed by the operating system.  Running a quick date command proved enough.  No redirection was necessary as we were already recording everything into a file.
• We needed to know what type of system we were running our commands on.  A quick uname accomplished this.  Of course different distros also have different methods of providing version information about themselves.  Look to website for the distro you are working with to determine which file to check in and then cat that information so that it is displayed to the screen and thereby written to your script file.
• In order to get all of the system information written to the correct directory we needed several commands.  Unfortunately, if the system had been compromised and a rootkit had been deployed, there was the potential that critical information would not be copied.  To help alleviate some concern behind this issue we decided to run a few commands.  First we needed to know where each of our commands were located.  This was determined through a simple which command.  This let us know where the system thought the program was located and allowed us to call it directly as we did not want to rely on the $PATH environment variable.  Next we needed to get the versions for each command so that we knew which versions of the commands we were working with in case of any problems or other issues such as non-standard programs.  Most commands will take either a –version or -V and provide specific details about the command.  In this case we needed to know the version of md5sum, dd, and split that was provided by the system.  Once this information was displayed we had it recorded.  Next we took the md5sum of each of these commands just to be sure.  If necessary these outputs could be checked against known good versions from known good sources of the distro.  If really paranoid and the tools are available, other hashing tools could be used.  Although all of these tools could be “rooted” it is not very likely.  Plus, in this instance, we were not too worried about a rootkit, but you never know.
• Now were were ready to copy the information to our mounted directory.  The hard drive was recognized as hda by the operating system.  This means that we needed to copy /dev/hda using the dd command.  But since we were copying a large hard drive (80 GB +) we needed to use the split command to help with errors.  Now, I did not know this before we started, but I heard from one of my co-workers and it was confirmed by the system administrator, that copying large files (everything is a file in *nix) to a NFS mounted directory was not completely reliable.  To help control this I was told that copying to 2GB files would be the best way to handle it.  If something failed then it would be easier to pick up from the last good file and just calculate the proper offsets to start the copying process from  that point instead of the beginning.  Here is the command we ended up running: /bin/dd if=/dev/hda | /usr/bin/split -d -b 2000m – /tmp/target/server_name.dd. Although you can review the split man page to understand the full command, I will point out two things.  The dash “-” is necessary because it tells the command to use the STDIN from the pipe as the input file.  The “.” at the end of the command is also necessary because it is what split uses to number the files it creates.
• Depending on the system this will run for quite a while.  But before you run off for coffee and donuts or to analyze another system you’ll want to run one more command.  That command is the date command.  To do this just type date and press enter.  This will automatically run the date command once the previous command has completed running.  This time and date gets recorded into the script and the full run time is recorded for use in reporting and future efforts.

Although all of this took some effort to do, the end result was that I had acquired the system in a manner that I could use multiple tools to evaluate the information.  Once this was all complete I moved onto the second webserver where we had similar issues with acquiring the hard drive as the system was running on the same blade server.  I would love to tell you that I just followed my previous steps and that all went well.  Unfortunately I cannot.  We did perform all of the steps but as soon as we hit enter on the dd command the system virtually locked up.  This came as quite a surprise as the previous system did not take a significant performance hit during the acquisition.  We stopped the command and did quick review of the logs on this new system.  The log was full of I/O errors and we could only assume that the hard drive was slowly failing.  Therefore, plan three == fail.  That is where the system administrator came to my rescue.  As the system did not have any problem backing up every night we decided to just tar the operating system over to the NFS mounted directory.  Although this would not get everything off of the drive, it would at least get some of the information and some is better than none, at least at this point. Here how the tar was accomplished.

• Follow all of the previous steps to get the information about the system and the commands.
• Next you need to understand the version of tar provided by the distro you are working with.  Read the man page just to be sure you have the options determined correctly.  Because of the way that tar interacts with NFS (thank you, schism) it is probably going to be necessary to use the “–one-file-system” option.  This means that you will have to acquire each of your mounted directories separately.  For instance, if you have one partition mounted at /, one at /boot, and one at /home the “–one-file-system” command will not follow down the mounted directories.  You will have to do each separately.  The following command worked like a charm and then repeated for each mount point.  /bin/tar –one-file-system -cvf /tmp/target/server_name_root.tar /

So, after four tries I had all of the information that I needed to start my analysis.  Well, at least for these systems.  This simple acquisition really threw me for a loop initially.  Not only did I have to pull out multiple acquisition techniques it also affected my time on site.  That might not seem like much, but these extra step meant that I was on site for an additionally two days.  No small impact on the customer.  But, it could have been a lot worse.  If I had not been prepared with fall back alternatives, then I could have spent even more time on site.  Not only would this have impacted the customer’s pocket book but it would have also sewn the seeds of doubt into my capabilities.  This could have, in turn, jeopardized their view of the results of the analysis as well as the potential for the continuation of a forensic, or any other, business relationship.

Go forth and do good things,

Don C. Weber

Jeremiah Grossman has a simple but sweet post about what to do on your first day of work when you come on board to a company that has no “no existing web/software security program.” He simply asked, “What is the very first thing do on day 1? [sic]”

The meat of the post is in the comments. Although it started out with some typical guidance on how to technically identify server, applications, vulnerabilities, and the like, the comments quickly transition into focus on the people of the organization. Getting to know the executives, management peers, security and technical administrators, and even support personnel before diving in and trying to find problems and giving orders about how to fix them.

Security Professionals need to remember that there are other people out there. It has often been said that we need to refrain from saying “No,” “Don’t,” “Can’t,” and other negatively connotative words unless absolutely necessary. We often remind ourselves that we are a part of the business unit and that we are, typically, support personnel rather than the front line administrators (and if you are both then your security tasks should take the support model into consideration). So when it all boils down, we are saying that we have to be a helpful and viable part of the business by working with the other employees, no matter the level, rather than being the lonesome cowboy with six-guns drawn. Once we have accomplished this then we can start delving into identify critical physical assets, location of data, mission critical application, and other important technically-related security information. Hopefully, your initial dealings with fellow employees and managers will have already greased the skids to start working with this information, but it will have also provided you with a better understanding of the politics and business necessities surrounding the current state of technical deployment.

I’m not going to repeat my or anybody else’s comments here. Go check out Jeremiah’s post and then put in your two cents. But while you are there, notice some of the names of people who are commenting on getting to know the people and organization first before diving into the technical aspect of the position. You will probably notice many people that you know and respect.

Go forth and do good things,

Don C. Weber

I would like to start off by saying, “You can’t!!” The quicker you come to grips with that the better off you will be in the long run. Politics, or perhaps Micro-Politics since I am talking about intra/inter-office politics, is just a fact of life. Everybody has an agenda whether it is to further themselves, further their family, further the company, or any number of other things. So, get over it because it is just going to happen.

Now, let me tell you how you can control politics. I’m not talking “hand of God” control. I’m talking about making it difficult for politics to adversely (because some politics are good) influence the security of your organization. The answer can be found in my previous post on Organized Security. The answer is “Document Your Processes!” Okay, that is not the full answer, but it is the start. Getting your processes written down and accepted is the first step. The thing that seems to be working the best for my team is to document a process’ flow before writing down the procedure. Understanding the actions, decisions, and touch points of a process before writing the document that details each action and decision point. Here is a simple example pertaining to a user account request. This process flow utilizes “swim lanes” to show different teams or departments.

Once you have created this flowchart it is very hard to justify a deviation from this process. It becomes even more difficult once you detail each box in your procedural documentation. Getting your management and each team or department listed in the “swim lanes” to sign off on their involvement with the process will decrease the deviation possibilities even more. And if all else fails, it will make deviations readily apparent to management and all of the teams or departments involved.

Now, this does not mean that deviations will not happen. It is a fact of life that a situation or event was not taken into consideration during the development of the process. These instances shouldn’t matter in the grand scheme. Once the event has happened and been addressed, the individuals responsible for the process should quickly run through the process to see if any documentation needs to be generated or additional actions taken. After everything has been addressed the team can conduct a lessons learn to determine if the process needs to be updated or if the deviation was just an anomaly that will rarely occur and can be addressed on a case by case basis. Of course, politics can fall into this category. But all of this, as I mentioned, makes the deviation very apparent and the extra work associated with running back through the process and evaluating the overall process should raise questions about the validity of the action.

Once everything is documented and approved there is another very important step. That step is to consistently apply the process. Lack of consistency will leave gaps in all of your processes. Lack of consistency will breed contempt for your system and provide individuals and groups the leverage they need to circumvent the process in question and possibility the other processes developed by your team.

In the end you are not going to solve politics in your organization. You and your team need to learn how to accept it as a part of doing business. Just remember, diligent documentation, repeatable processes, and consistent application will protect you as much as they can.

Go forth and do good things,

Don C. Weber