Security Ripcord

Slow week, again.  My wife is loving it that I have been at home recently.  It’s good to be wanted.

Leadership

I have been plowing through the Enderverse over the past month.  My wife finds it weird how I get fixated on reading.  I am not surprised that it is new to her because I have not really read a non-technical book since completing my Master Degree in 2005.  The whole time she has known me the only thing she has really seen me get fixated on (other than task completion) is moving through a new video game.  I guess she should have seen the signs.

Reading the full Ender and Bean series has helped me understand Ender’s Game a lot better.  These books show the importance of understanding yourself, understanding the people you work with, understanding the people you work against, strategy, and many other leadership qualities.  Certainly you can get these from other books, but the Enderverse does a good job of showing enough of both sides of the story to understand motivation.  Which I am finding to be the biggesst take-away from this series.

Orson Scott Card demonstrates how effective it is when leaders understand an individual’s motivation.  Actually, it goes a little deeper that than.  He has his main and most successful characters leading by understanding what the people around them actually “need.”  They take those needs into consideration for their own plans and actions.  Whether it is a personal relationship or it is military strategy.  These characters are most successful when they step back, take a moment for consideration, and modify their plans and judgments (when necessary) to accommodate the needs of the people around them.  Doing so usually wins that character the professional respect, friendship, and sometimes devotion.

How can we use this in our professional and personal lives?  Well, I think the best away to do this is to take a few seconds before answering questions or providing opinions during conversations (live or digital).  A few extra seconds pause during a phone call is not as long as it actually seems on your end.  Pauses are natural in face-to-face conversations because it is easy to see when somebody is thinking.  If people on the other end of the line do get impatient, just ask for a second while you are thinking or formulating a response.  It will be a rare occasion when somebody gets offended and it will be more likely that they respect you for being considerate and thoughtful.  Email and IM is another great example for a pause.  Get up, walk around, and don’t answer until you have given a little thought to your position in combination with the other person’s needs.    You can even do these types of things in meetings and one-on-one conversations.  Taking a few moments to lean back and ingest the facts of the conversation is easy and common.  For heated conversations or times when you need more time to think about it, use the bio-break method to get some extra time.  Get up, go to the bathroom, get a glass of water, and think about your position, your stratagy, and weight that with the needs of others involved in the situation.  It doesn’t matter if others know you are just using it as an excuse, nobody is going to deny anybody a bio-break.

Quick Tip

I have been working on a bit of code for several months now.  I am getting into the fine-tuned debugging and I hit a snag.  One piece of advice that the mentor for my project gave me was “never be afraid to revisit and rewrite your code from scratch.”  Well, I am not there yet.  But I am to the point where I understand that I may have made a mistake with my implementation and I need to determine how much I need to change.  Of course, now that the code is at 1855 lines (with comments) it is a little harder to scroll back and forth to find all of the areas that need to be adjusted for the implementation change.

I have decided to fall back to printing and editing by pencil.  It should not be too hard but 1855 lines of code divided by 80 lines per page is 23 pages of code.  Ouch.  Of course there are easier ways to do this.  People have been writing and printing code for years and there are all types of tips and tricks.  *nix OSes are naturally full of different methods for converting text to printer and other document formats.  (Now that I think about it that sounds like the potential for vulnerable programs and scripts is huge….hmmm…but I digress).  Actually I knew this already and I use to have a nice little script that would print two pages per sheet, in landscape, and modify the font so that it was small but not too small to read and edit.  A quick look showed me that I had disposed of that code a long time ago (several OSes ago).  But, the Internet is rich with ideas and methods.  I ran through some instances of enscript (not to be confused with EnCase’s scripting language) which seems to be the defacto method nowadays.  But for some reason it just wasn’t working for me.  AND it was not the method I used before (we are all creatures of habit) so I continued my search until I found my one true-code-printing-love: a2ps. A quick review of the very helpful man page gave me the syntax I needed to convert the file I want to print to a postscript file  (my computer is not connected to a printer right now and I have to use a friend’s).  I ended up with the following command line (I will let you review the man page to understand the options):

a2ps -2rjC -Epython -o ptr-file.ps ptr-file.py

Easy, but what if my friend’s system cannot read Postscript files?  Doubtful but just to be certain I decided to quickly convert to PDF via command line.

ps2pdf ptr-file.ps ptr-file.pdf

Yes, yes.  It would have been much easier to combine these two commands.

a2ps -2rjC -Epython ptr-file.py | ps2pdf – ptr-file.pdf

Enjoy.

Personal Input

Read more non-technical books.  It is good for you and it is good for your kids (if you have them) to see you reading instead of watching TV, or cleaning the house, or doing yard work, or playing with them, or……dang it!!!! 

Go forth and do good things,

Don C. Weber

Slow week, lots of programming.  Gonna rain this weekend.  Looks like we’ll be hanging out at the library and aquarium.

Leadership

During my bike ride on Monday I heard a blast from the past.

“Raise Your Voice” – Bad Religion

fa fa fafa fa fa fafa Raise Your Voice!
Don’t be played like someone else’s board game
Don’t be classed out like some desolate redoubt
Don’t be misled you’ve got alot on your head
And nobody’s gonna pay attention when you are dead
So: fa fa fafa fa fa fafa Raise Your Voice!
It’s the primary rule, you gotta wanna be fooled
It’s our daunted restraint that keeps us silent in shame
It’s our nature to be adversarial and free
Our evolution didn’t hinge on passivity
fa fa fafa fa fa fafa Raise Your Voice!

Actually, the song from the album begins with the lead singer stating “I think this is song we could redo in every language of every country we go to.”  True.  I think that every security professional can learn a little from these lyrics as well.

What role do security professionals play?  Auditor, monitors, engineers?  Actually, I think we are just members of the team fulfilling specific roles to keep our organization operating smoothly.  But we cannot do so in silence.  We cannot just push out reports, scan  outputs, and presentations on how people and the organization should operate.  These things are too easy to dismiss.  No, we have to do the thing that some of us dread and many more people don’t want to happen.  We need to “raise our voice.”  We need to talk about the issues.  Have open discussions that challenge the status-quo.  We need to confront common issues with new ideas and methods.

What do we accomplish when we are doing this?  Are we trying to get our way no matter what the cost?  No.  We are trying to open minds and continuously adapt and evolve.  We are verbally raising issues so that the conversations lift from the paper and actively carry on in other discussions.  We are forcing people to actively observe the current situation, address our recommendations, accept or counter these recommendations, and improve themselves and the organization.

One trap that is easy to fall into, however, is thinking that we are always in the right.  That our ideas are the correct direction.  I challenge you to “raise your voice” while also keeping your mind open to new information and other possibilities.  I challenge you to find a way to persuade those that don’t understand while also realizing that you may be one of those who does not have a clear vision of the full picture.  I challenge you to be confident yet humble in your expertise and your efforts to improve your organization and yourself.

I challenge you to “raise your voice,” no matter who is in the room.

Training

No training developed this week.  I did, however, take a few minutes to take last weeks quick tip on HBGary’s Fast Dump Pro and turned it into a training document for the rest of the analysts in my team.  Although the tool is pretty easy to use having an internal document for some of the tools that you don’t use on a regular basis is very helpful.  This allows you to quickly re-familiarize yourself with the tool.  It also helps train new personnel to the methodologies of the team.

How can you contribute to your team?  Isn’t there something that you could quickly write up that would benefit everybody?  If you don’t have that type of system in your team, would creating one help?

“Raise your voice.”

Quick Tip

You will need Mark Hammond’s pywin32 and Tim Golden’s wmi.py for this tip.  Basically, I want to show you an easy way to get exactly the system information that you want.  It will be easy to get all of the system information using Win32_OperatingSystem.  Using Tim’s method you can directly access the specific fields that you want. (inserted periods to help with proper indention due to Wordpress stripping whitespace and my need for Python-structure OCD)

import wmi
c = wmi.WMI()
for os in c.Win32_OperatingSystem():
…print os.Caption

But this means that you either have to request each on individually.  This is not so bad and easy if you just want a few specific items.  But what if you want more than a few bits of information?  And what if Microsoft changes the information provided across systems?  You want to be sure that you can access that information without the script failing on you.

To handle this I suggest using a tuple to hold the values of the fields used by Win32_OperatingSystem.  Then run through the tuple using hasattr and getattr to pull the information provided by your call to WMI for the Win32_OperatingSystem information.  Here is an example script .  Of course you don’t have to include everything (as I have done here for clarity).  You can select the fields that are most important to you.  Or, you can include them all and comment out the ones you don’t want.  That way they are easily added in the future when you discover a need.

import wmi

Win32_OperatingSystem_Fields = (
‘BootDevice’,
‘BuildNumber’,
‘BuildType’,
‘Caption’,
‘CodeSet’,
‘CountryCode’,
‘CreationClassName’,
‘CSCreationClassName’,
‘CSDVersion’,
‘CSName’,
‘CurrentTimeZone’,
‘DataExecutionPrevention_Available’,
‘DataExecutionPrevention_32BitApplications’,
‘DataExecutionPrevention_Drivers’,
‘DataExecutionPrevention_SupportPolicy’,
‘Debug’,
‘Description’,
‘Distributed’,
‘EncryptionLevel;’,
‘ForegroundApplicationBoost’,
‘FreePhysicalMemory’,
‘FreeSpaceInPagingFiles’,
‘FreeVirtualMemory’,
‘InstallDate’,
‘LargeSystemCache’,
‘LastBootUpTime’,
‘LocalDateTime’,
‘Locale’,
‘Manufacturer’,
‘MaxNumberOfProcesses’,
‘MaxProcessMemorySize’,
‘MUILanguages’,
‘Name’,
‘NumberOfLicensedUsers’,
‘NumberOfProcesses’,
‘NumberOfUsers’,
‘OperatingSystemSKU’,
‘Organization’,
‘OSArchitecture’,
‘OSLanguage’,
‘OSProductSuite’,
‘OSType’,
‘OtherTypeDescription’,
‘PAEEnabled’,
‘PlusProductID’,
‘PlusVersionNumber’,
‘Primary’,
‘ProductType’,
‘RegisteredUser’,
‘SerialNumber’,
‘ServicePackMajorVersion’,
‘ServicePackMinorVersion’,
‘SizeStoredInPagingFiles’,
‘Status’,
‘SuiteMask’,
‘SystemDevice’,
‘SystemDirectory’,
‘SystemDrive’,
‘TotalSwapSpaceSize’,
‘TotalVirtualMemorySize’,
‘TotalVisibleMemorySize’,
‘Version’,
‘WindowsDirectory’
)

class sysWMI():
…def __init__(self):
……self.wmiObj = wmi.WMI()

…def getSysInfo(self):
……info = {}
……for obj in range(len(Win32_OperatingSystem_Fields)):
………if hasattr(inf, Win32_OperatingSystem_Fields[obj]):
…………print Win32_OperatingSystem_Fields[obj] + “: ” + str(getattr(inf, Win32_OperatingSystem_Fields[obj]))

sysInfo = sysWMI()
sysInfo.getSysInfo()

Personal Input

Happy Birthday #4, Collier!!!   I love you, son.  You are a great son and I am proud to be your father. Although sometimes I raise my voice TO you, I raise my voice FOR you everyday.

Go forth and do good things,

Don C. Weber

Andy Willingham, or Andy ITGuy as some of you know him, posted an interesting question in the Security Catalyst forums the other day in a post titled “Is it really ‘Game Over’?”

I attended GFIRST this week and many of the presentations gave the impression that we have hit to point where it’s “game over”. None of them ever said that exactly but Amit Yoran of NetWitness came pretty close. He said that we have already lost and it’s going to get worse before it gets better.

What are you seeing in your “security life”?
Is it that bad? Is it almost that bad?
What do you think we need to do to change the course of things?

… (Go see his opinion in the post)

Now, I haven’t been contributing to the Security Catalyst forums recently (although I know I should be), but I couldn’t let this one slip by without a comment.  Basically I take the stand point that there is no “Game Over” or even “Win or Lose” in the information technology security industry.  There is only “Adapt and Then Evolve.”   Some people might consider this “Playing Catch-up” and it can seem like it if you want to take a pessimistic attitude towards it (we all do multiple times during our careers).  I, however, view it as a constant struggle like life.  We aren’t here to “Win or Lose.”  There is not an “End Game” scenario.  There is only the constant struggle to survive the best way that we can and, while we are at it, make sure that good people are not preyed on by the scumbags in every society.  If we are very lucky, we will also make enough to support our families comfortably while they are dealing with their life struggles.

I have provided a few more details in my reply to Andy’s forum post.  Go, read it, chew on it a bit, then provide your input so that others can learn and grow.  After all, by doing so you are helping others “Adapting and Evolving” by being a Security Catalyst.

Go forth and do good things,

Don C. Weber

Okay, I have been out of the blogging game a little too long.  It is time for me to start generating some more content.  So we will begin by doing a weekly wrap-up.  Basically I am going to go over a few of the things that occurred during my week.  I might even just set up a draft and populate as I go the hit submit on Friday afternoon.  As everybody will have started drinking by then my post should be well received.

I’m going to start out a little formless.  Hopefully this will start taking a  little better shape as we move forward.

Leadership

Lead from the front.  Some people might not like it, but the only reason they have time to bitch about it is because they are not doing it themselves.  You only learn from your mistakes.  If you aren’t leading you aren’t making enough mistakes to challenge yourself and provide yourself with enough opportunities to improve.

Training

I finally got around to using Wink the other day.  I developed some training using the same old Power Point slides everybody seems to generate.  You know the type, all words and no joy….er…pictures.  Even when you have a screen shot in PPTs it is just more words.  So, at the request of one of my team, I branched out with some flash video.  Of course Irongeek has been doing this for years and provides a walk through of what and how he does it: How I make Hacking Illustrated Videos.  Luckily I didn’t have to refer to Irongeek’s site as Wink was very easy to use.  I generated a quick training on doing a network capture using tcpdump.  Unfortunately I did it on my work system so I cannot provide it here.  But the point I wanted to make was that it was very easy.  Secondly, after pulling the video off my Linux system and onto my Windows system for emailing to the team, I noticed that I could import the Wink project back into the Windows version and add audio.  Now, I haven’t tried this completely, but it seemed straight forward.  So, hopefully, we will be seeing some of this later.  Next time I’ll generate the video on my own box and I’ll be able to provide it here.  Besides, everybody already knows how to do network capture with tcpdump, right?

Quick Tip

For the training I was just talking about I had to create a 10MB file.  It didn’t require any real data, I just needed some random bits but in a large file for demonstrating the file splitting functionality of tcpdump.  So I tried using “dd” to fill up the 10MB file using the following command.

# dd if=/dev/random 0f=./test_10MB.dat

I was doing this on a Ubuntu Linux box.  Although this seems like it would work, and there are plenty of search results that say to do it using this technique, this does not work on Ubuntu Linux.  All this does is create a 4KB file.  I tried all kinds of different concatenation techniques.  But I couldn’t get a 10MB file.

It turns out that Linux systems do not produce enough entropy to fill a 10MB file quickly.  If the Wikipedia /dev/random entry is to be trusted we see that “When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered. (Source: Linux Programmer’s Manual, section 4)”.

So, left out to dry I queried a friend of mine (Thank you, Schism) and he pointed out that the best way to do this is to use the following command.

# dd if=/dev/urandom 0f=./test_10MB.dat

Which, of course, works like a charm.  He was also quick to point out that this does not provide the “randomness” as is produced by /dev/random, but for my purposes it worked well enough.

Development

I have been working with Python programming a bit over the last few months.  It has been interesting watching the program I have been working transform into something useful.  I wish that I could provide a little more information about it, but it is still an ongoing project, so I’ll refrain.

One thing I can talk about is building function calls in Python.  I am not sure if I am using the right terminology for it, but basically what I mean is creating a list of functions and then calling them based on the results from some other function.  For instance if you can define a table or list of functions like this:

func_table = ( one, two, three )

Then create a function of those names (necessary indent removed by Wordpress, so just pretend)

def one():

print “one”

def two():

print “two”

def three():

print “three”

Finally you can call each function by referencing them via the list.

for i in range (3):

func_table[i]()

When run this will produce

one

two

three

Special note: if you are returning information from these functions then you will need to store them in a variable before using or returning.  At least that is my experience.

Not very exciting until you start thinking about using tests instead of iterating through the loop.  This can be used to clean up complex code very nicely.  My understanding is that this will also help with optimizing the execution of the code as well.  What the magic balance between this method and several “if” statements is, I do not know.

Can anybody describe to me a good method of testing this optimization?  I don’t know enough about programming to come up with a complex enough task to challenge the CPU and memory of my system and produce viable results.  If you have a recommendation leave a comment and I will test and post results.  Or, you can run it yourself and post your results in the comments.  Either way it will be beneficial to us all.

BTW, thank you, Invisigoth.  I learned this from reviewing your code.

Personal Input

Not much today as I need to get down stairs with the boys.

Go forth and do good things,

Don C. Weber

Following up on any project is key.  Talking to all involved about what has happened, why it happened, how it could have better, what worked very well, etc is the key to improvement.  So, why aren’t organizations that experience a security related incident able to prevent or curtail future incidents?  Basically because they are not following up the incident with a lessons learned, or they are not doing it properly.

Incident Response should be handled like any other project.  It should be managed.  As most incident responders are aware, SANS GCIH course outlines 6 distinct phases of an incident response:

1. Preparation
2. Identification
3. Containment
4. Eradication
5. Recovery, and
6. Lessons Learned

Ah, Lessons Learned is the final step.  Actually, it is circular, so the final step leads into the first step.   I say leads into whereas a better metaphor might be “feeds.”  Or in the terms of following flow charts, for you management types out there, the outputs of Lessons Learned are the inputs of Preparation.  Here’s a question for you though, during your preparation phase did you look at the methodologies that you use to improve how you identify, prioritize, address, and follow-up on your lessons learned?  Or, when you are finished with your lessons learned meeting, do you have a list of action items that have been assigned to a specific individual who understands the criteria for successfully completing the action?

I just asked two questions that took your simple lessons learned meeting from a quick five minute session to a thirty minute plus session.  Although you might think that the issues will drive how long this meeting will last, in actuality that is not correct.  With proper methodology, in other words a practiced plan, this meeting can still be very quick while producing the key outputs that are necessary to augment your preparation phase.

Silver Bullet time.  Yes, I know, you want the silver bullet that is going to help you increase the effectiveness of your lessons learned process.  Guess what!!  Most likely you already have it.  Do you use process management to plan your software, hardware, or infrastructure development?  Bingo, then you have the means to improve your lessons learned.  Start using the processes that you already have in place.  I say this because it is the fastest and cheapest method to gaining control over this process.

For those of you who do not have a process in place, never fear, I have one word for you: Why.  Just ask why.  But ask it five times.  Asking why five times is the technique for determining root cause.  Gasp, Root Cause Analysis.  It is used during the Six Sigma process as well as being integrated into other project development schemes.  In his 5 Whys post Eric Ries explains that Taiichi Ohno of Toyota fame wrote about this technique in his book Toyota Production System: Beyond Large-Scale Production.

Now, this might seem silly to some at first.  It will seem especially silly to those individuals and groups that are not use to management their projects or doing root cause analysis.  This is where the longer meetings are going to come into play.  Of course, this is true of any new process.  New things take time to understand and get use to performing.  For some people the learning curve on how to conduct themselves in these meetings is going to be a long and tough journey.  But by consistently applying one of these methodologies to your lessons learned process you will find that each of your meetings is shorter and more productive.  The implementors will be happier because they are being heard (if they are participating) and the managers and executives will be happier because of the increased productivity and effectiveness of the end results.

Certainly I have only touched on this topic briefly.  If you have techniques that have worked to improve the effectiveness of your lessons learned meetings, please share them with us in the comments.

Go forth and do good things,

Don C. Weber

http://www.sans.org/?utm_source=web&utm_medium=text-ad&utm_content=affiliate_link1&utm_campaign=Cutaway_Security

Not all incident responses go according to plan.  This means that a responder has to have multiple methods for accomplishing tasks.  Harlan Carvey brings this up on his blog often and it is why Ed Skoudis created his famous WMIC Kung Fu write-up and turned it into a SANS Course: Windows Command-Line Kung Fu In-Depth for Info Sec Pros.

On a recent engagement this fact came into play.  I went on site knowing that I would have multiple types of Unix and Linux operating systems to potentially acquire.  From the customer’s description I also knew that I would be dealing with some older equipment but nothing from their descriptions seemed out of the ordinary.  Once I  got on site I reviewed the situation with the customer and everything seem normal.  As I was not worried about collecting volatile data from the system I requested the customer set up a maintenance window and temporarily take the systems off line in a manner that provided the least amount of impact to their operations.

All went well until the system administrator handed me the hard drive.  It all seemed fine at first until I tried to plug the drive into the adapter that was connected to my write-blocker.  The pins didn’t match up.  Although there were plenty of pins across, the layers were placed too close together.  It was fine for the blade server that it came out of, but none of my adapters could plug into this configuration.  At first I figured this was no big deal.  I decided to boot to a Linux boot disk and just copy the drive from there to a USB hard drive.  Well, there was two problems with that: 1. The blade server did not have a CD-ROM drive, and 2. the USB port was version 1.1 and would not recognize the external hard drive that I had purchased (no big deal because 12MB per second would not cut it either!!).  So, plan one == bust and plan two == bust.

So what was plan three?  Well, dump over NFS using native commands, of course.  Luckily the system administrator was well versed in NFS and the commands we were about to use.  So there was not a lot of explaining, I didn’t have to type commands, and my only real job (as it should be in this type of situation) was mindful and guided keystroke monitoring.  Here is what we did.

• To start off, please review this great guide on setting up NFS via the Ubuntu community wiki.  It does not dive deeply into NFS.  It just gives the basics on how to get it up and running on a live system or Live CD.  Of course, mileage may vary depending on distro, so beware.  Once it was all set up we navigated to the correct directories and prepared to acquire.  When all was done the system we were acquiring had a remote share mounted to /tmp/target.
• Before we started copying files we needed to keep track of everything we did.  Of course we could have done that as soon as we logged into the system, but as it was the system administrator performing the actions and this was not necessarily a “Live Acquisition” it was not completely necessary and we needed to have a good place, like an NFS mounted directory, to copy our files.  So we changed to the /tmp/target directory and ran the script command to record our actions and sent it to an appropriately titled file.
• Next we needed to record the system time.  Although we did note the BIOS time when the system was booting up, we need to record the time displayed by the operating system.  Running a quick date command proved enough.  No redirection was necessary as we were already recording everything into a file.
• We needed to know what type of system we were running our commands on.  A quick uname accomplished this.  Of course different distros also have different methods of providing version information about themselves.  Look to website for the distro you are working with to determine which file to check in and then cat that information so that it is displayed to the screen and thereby written to your script file.
• In order to get all of the system information written to the correct directory we needed several commands.  Unfortunately, if the system had been compromised and a rootkit had been deployed, there was the potential that critical information would not be copied.  To help alleviate some concern behind this issue we decided to run a few commands.  First we needed to know where each of our commands were located.  This was determined through a simple which command.  This let us know where the system thought the program was located and allowed us to call it directly as we did not want to rely on the $PATH environment variable.  Next we needed to get the versions for each command so that we knew which versions of the commands we were working with in case of any problems or other issues such as non-standard programs.  Most commands will take either a –version or -V and provide specific details about the command.  In this case we needed to know the version of md5sum, dd, and split that was provided by the system.  Once this information was displayed we had it recorded.  Next we took the md5sum of each of these commands just to be sure.  If necessary these outputs could be checked against known good versions from known good sources of the distro.  If really paranoid and the tools are available, other hashing tools could be used.  Although all of these tools could be “rooted” it is not very likely.  Plus, in this instance, we were not too worried about a rootkit, but you never know.
• Now were were ready to copy the information to our mounted directory.  The hard drive was recognized as hda by the operating system.  This means that we needed to copy /dev/hda using the dd command.  But since we were copying a large hard drive (80 GB +) we needed to use the split command to help with errors.  Now, I did not know this before we started, but I heard from one of my co-workers and it was confirmed by the system administrator, that copying large files (everything is a file in *nix) to a NFS mounted directory was not completely reliable.  To help control this I was told that copying to 2GB files would be the best way to handle it.  If something failed then it would be easier to pick up from the last good file and just calculate the proper offsets to start the copying process from  that point instead of the beginning.  Here is the command we ended up running: /bin/dd if=/dev/hda | /usr/bin/split -d -b 2000m – /tmp/target/server_name.dd. Although you can review the split man page to understand the full command, I will point out two things.  The dash “-” is necessary because it tells the command to use the STDIN from the pipe as the input file.  The “.” at the end of the command is also necessary because it is what split uses to number the files it creates.
• Depending on the system this will run for quite a while.  But before you run off for coffee and donuts or to analyze another system you’ll want to run one more command.  That command is the date command.  To do this just type date and press enter.  This will automatically run the date command once the previous command has completed running.  This time and date gets recorded into the script and the full run time is recorded for use in reporting and future efforts.

Although all of this took some effort to do, the end result was that I had acquired the system in a manner that I could use multiple tools to evaluate the information.  Once this was all complete I moved onto the second webserver where we had similar issues with acquiring the hard drive as the system was running on the same blade server.  I would love to tell you that I just followed my previous steps and that all went well.  Unfortunately I cannot.  We did perform all of the steps but as soon as we hit enter on the dd command the system virtually locked up.  This came as quite a surprise as the previous system did not take a significant performance hit during the acquisition.  We stopped the command and did quick review of the logs on this new system.  The log was full of I/O errors and we could only assume that the hard drive was slowly failing.  Therefore, plan three == fail.  That is where the system administrator came to my rescue.  As the system did not have any problem backing up every night we decided to just tar the operating system over to the NFS mounted directory.  Although this would not get everything off of the drive, it would at least get some of the information and some is better than none, at least at this point. Here how the tar was accomplished.

• Follow all of the previous steps to get the information about the system and the commands.
• Next you need to understand the version of tar provided by the distro you are working with.  Read the man page just to be sure you have the options determined correctly.  Because of the way that tar interacts with NFS (thank you, schism) it is probably going to be necessary to use the “–one-file-system” option.  This means that you will have to acquire each of your mounted directories separately.  For instance, if you have one partition mounted at /, one at /boot, and one at /home the “–one-file-system” command will not follow down the mounted directories.  You will have to do each separately.  The following command worked like a charm and then repeated for each mount point.  /bin/tar –one-file-system -cvf /tmp/target/server_name_root.tar /

So, after four tries I had all of the information that I needed to start my analysis.  Well, at least for these systems.  This simple acquisition really threw me for a loop initially.  Not only did I have to pull out multiple acquisition techniques it also affected my time on site.  That might not seem like much, but these extra step meant that I was on site for an additionally two days.  No small impact on the customer.  But, it could have been a lot worse.  If I had not been prepared with fall back alternatives, then I could have spent even more time on site.  Not only would this have impacted the customer’s pocket book but it would have also sewn the seeds of doubt into my capabilities.  This could have, in turn, jeopardized their view of the results of the analysis as well as the potential for the continuation of a forensic, or any other, business relationship.

Go forth and do good things,

Don C. Weber

Jeremiah Grossman has a simple but sweet post about what to do on your first day of work when you come on board to a company that has no “no existing web/software security program.” He simply asked, “What is the very first thing do on day 1? [sic]”

The meat of the post is in the comments. Although it started out with some typical guidance on how to technically identify server, applications, vulnerabilities, and the like, the comments quickly transition into focus on the people of the organization. Getting to know the executives, management peers, security and technical administrators, and even support personnel before diving in and trying to find problems and giving orders about how to fix them.

Security Professionals need to remember that there are other people out there. It has often been said that we need to refrain from saying “No,” “Don’t,” “Can’t,” and other negatively connotative words unless absolutely necessary. We often remind ourselves that we are a part of the business unit and that we are, typically, support personnel rather than the front line administrators (and if you are both then your security tasks should take the support model into consideration). So when it all boils down, we are saying that we have to be a helpful and viable part of the business by working with the other employees, no matter the level, rather than being the lonesome cowboy with six-guns drawn. Once we have accomplished this then we can start delving into identify critical physical assets, location of data, mission critical application, and other important technically-related security information. Hopefully, your initial dealings with fellow employees and managers will have already greased the skids to start working with this information, but it will have also provided you with a better understanding of the politics and business necessities surrounding the current state of technical deployment.

I’m not going to repeat my or anybody else’s comments here. Go check out Jeremiah’s post and then put in your two cents. But while you are there, notice some of the names of people who are commenting on getting to know the people and organization first before diving into the technical aspect of the position. You will probably notice many people that you know and respect.

Go forth and do good things,

Don C. Weber

I would like to start off by saying, “You can’t!!” The quicker you come to grips with that the better off you will be in the long run. Politics, or perhaps Micro-Politics since I am talking about intra/inter-office politics, is just a fact of life. Everybody has an agenda whether it is to further themselves, further their family, further the company, or any number of other things. So, get over it because it is just going to happen.

Now, let me tell you how you can control politics. I’m not talking “hand of God” control. I’m talking about making it difficult for politics to adversely (because some politics are good) influence the security of your organization. The answer can be found in my previous post on Organized Security. The answer is “Document Your Processes!” Okay, that is not the full answer, but it is the start. Getting your processes written down and accepted is the first step. The thing that seems to be working the best for my team is to document a process’ flow before writing down the procedure. Understanding the actions, decisions, and touch points of a process before writing the document that details each action and decision point. Here is a simple example pertaining to a user account request. This process flow utilizes “swim lanes” to show different teams or departments.

Once you have created this flowchart it is very hard to justify a deviation from this process. It becomes even more difficult once you detail each box in your procedural documentation. Getting your management and each team or department listed in the “swim lanes” to sign off on their involvement with the process will decrease the deviation possibilities even more. And if all else fails, it will make deviations readily apparent to management and all of the teams or departments involved.

Now, this does not mean that deviations will not happen. It is a fact of life that a situation or event was not taken into consideration during the development of the process. These instances shouldn’t matter in the grand scheme. Once the event has happened and been addressed, the individuals responsible for the process should quickly run through the process to see if any documentation needs to be generated or additional actions taken. After everything has been addressed the team can conduct a lessons learn to determine if the process needs to be updated or if the deviation was just an anomaly that will rarely occur and can be addressed on a case by case basis. Of course, politics can fall into this category. But all of this, as I mentioned, makes the deviation very apparent and the extra work associated with running back through the process and evaluating the overall process should raise questions about the validity of the action.

Once everything is documented and approved there is another very important step. That step is to consistently apply the process. Lack of consistency will leave gaps in all of your processes. Lack of consistency will breed contempt for your system and provide individuals and groups the leverage they need to circumvent the process in question and possibility the other processes developed by your team.

In the end you are not going to solve politics in your organization. You and your team need to learn how to accept it as a part of doing business. Just remember, diligent documentation, repeatable processes, and consistent application will protect you as much as they can.

Go forth and do good things,

Don C. Weber

Well, this week at work was very interesting. Actually, the last two weeks have been extremely busy. As Friday rolled in I looked into the eyes of my team members and I could see the tired, slightly overwhelmed, and, for some, haggled look in their eyes. They had shouldered what our organization decided to throw at them and they pulled through with their heads held high. No small feat when you are talking about a crew with that was built from individuals with very little security background and a manager (me) who is hell-bent on documenting and improving each procedure as they are going through it. I do this not only to help them build a program that is repeatable and lends itself to self-improvement, but so that our customer can “feel-the-pain” when their goals are not being accomplished due to the never ending “high priority” additional tasks (something I, and others, refer to as “firefighting”).

I usually make it a point to congratulate my team members for a job well done. It builds confidence, denotes achievement, and helps give a sense of closure to on-going tasks and issues that never seem to have an “end.” But this week I went a step further. I let them know that when they are working on the “high priority” issues, when the “firefighting” is taking all of their time and effort, that the things they are doing are enough. Just working the task is enough to help secure our environment. Even if they haven’t completed the task or specific issues mean they were not able to address regular duties and other tasks, as long as they worked hard and smart, it is enough.

It has to be enough. No environment is ever going to be 100 percent secure. Security professionals and security cynics can all agree to that statement. But, when you look at it from the other end, no environment is zero percent secure either. Each operating system comes with some controls. So every environment starts a little bit “in the black.” As an organization starts adding personnel and controls they increase their security percentage. Finally, with the addition of security professionals and a well-rounded security approach, an organization sees its greatest jump towards the unobtainable 100 percent secure goal. Just dong things to move towards that endpoint is enough. And I think that sometimes organizations and managers forget that aspect of the big picture.

So, when you get back in the office next week, take a look around. Look at the accomplishments of your team members. Take note of these accomplishments and provide the appropriate praise to the situation. Let them know that their efforts are enough and that because of their actions the overall environment is more secure. Then look at the other individuals in your organization. Look at the system administrators, the desktop support personnel, the help desk operators, and everybody else. Look at their actions and point out their accomplishments as well. Let them know that they are helping secure the environment and that their actions are enough.

If you do this, you are doing enough and you are speeding up your progress towards that unobtainable goal.

Go forth and do good things,

Don C. Weber

I’ve been doing a little running lately getting ready for the Corpus Christi Beach to Bay Relay. Today, instead of our usual four mile run, we decided to work on some sprints. We ran a mile and then started a series of 100 yard sprints with a 100 yard walk in between. Needless to say that the walking reset was filled full of huffing and puffing. At one point I noticed that I was hanging my head like most people do when they are tired. When I realized this I did what I always do, what I taught myself in the Marines after long runs and forced marches, I raised my head and started looking around. I use to do this because whenever you are the most tired is when you are the most vulnerable. You are not paying attention, you are breathing heavy, and you are doing everything you can just to take a break for a minute or two. Fortunately, the repercussions of me doing this now are not the same as they were back then.

All of this got me thinking about how we react to situations as a whole. I started thinking about how through training and effort we can begin to overcome hardships. I started thinking about how diligent practice can instill good habits and create muscle memory in any individual. Muscle memory is a condition where a body reacts without, or more precisely with only a little, thinking. You can see this by reviewing Rich Mogull’s posts on how he handled several car accidents after being out of the paramedics for a while. Rich did what came natural to him. He just reacted and, I’m sure, did a great job and a service.

“Yes, yes,” you are thinking to yourself right now. We have heard this all before. Practice makes perfect. Practice your incident response. Practice your backup procedures. Practice your disaster recovery. Practice makes perfect. Practice, Practice, Practice. Blah, blah, blah. Yes, I am tell you that. But what I want to emphasize is that you can train yourselves all day long and still make mistakes.

Running with my head down took me back to the days of running through the hills of Camp Pendleton and training myself to keep my head up and aware of my surroundings no matter how tired I was at the time. But what it really got me thinking about was being in the stack. Not the stack you are use to hearing about, the stack of Marines that are just about to enter a building or room that may contain hostiles. It didn’t matter where we were, once people started lining up and getting ready to move to action, their heads dropped. Not because they were tired or lazy, but because they were focused and waiting. Like a spring ready to uncoil all of its power. This occurred so often that it was not surprising to hear, “Keep your heads up in the stack!” whispered over the radio. Or have someone give you a quick rap on the helmet as a reminder. Everybody did it, everybody got sucked into it, and everybody was aware of it and watched out for their buddy, because that person was watching out for them.

So, how does this apply to us? Well, security professionals have a lot to accomplish on any given day. Logs to review, servers to patch, incidents to respond to, training to develop and give (and that is just the short list). Let’s face it. We are swamped with responsibility and duties. Everybody groans when we walk into a room but everybody notices when our duties start falling behind because it directly affects their business. With all of this activity, with all of this responsibility, it is very easy to get set into a common routine or mode. It is very easy for our heads to drop into our computers, logs, management consoles, spreadsheets, etc. We are doing our jobs and we are getting it done, but are we aware of our surroundings. Are we aware of the common sights and sounds of the office environment and server room. Are we listening to people talk when they need our guidance, input, or for us to listen for listening’s sake?

If you are, then good on you. Now look around and see who is not. Please, tap them on the head and tell them, “Keep your head up in the stack!”

Go forth and do good things,

Don C. Weber