Security Ripcord

Not all incident responses go according to plan.  This means that a responder has to have multiple methods for accomplishing tasks.  Harlan Carvey brings this up on his blog often and it is why Ed Skoudis created his famous WMIC Kung Fu write-up and turned it into a SANS Course: Windows Command-Line Kung Fu In-Depth for Info Sec Pros.

On a recent engagement this fact came into play.  I went on site knowing that I would have multiple types of Unix and Linux operating systems to potentially acquire.  From the customer’s description I also knew that I would be dealing with some older equipment but nothing from their descriptions seemed out of the ordinary.  Once I  got on site I reviewed the situation with the customer and everything seem normal.  As I was not worried about collecting volatile data from the system I requested the customer set up a maintenance window and temporarily take the systems off line in a manner that provided the least amount of impact to their operations.

All went well until the system administrator handed me the hard drive.  It all seemed fine at first until I tried to plug the drive into the adapter that was connected to my write-blocker.  The pins didn’t match up.  Although there were plenty of pins across, the layers were placed too close together.  It was fine for the blade server that it came out of, but none of my adapters could plug into this configuration.  At first I figured this was no big deal.  I decided to boot to a Linux boot disk and just copy the drive from there to a USB hard drive.  Well, there was two problems with that: 1. The blade server did not have a CD-ROM drive, and 2. the USB port was version 1.1 and would not recognize the external hard drive that I had purchased (no big deal because 12MB per second would not cut it either!!).  So, plan one == bust and plan two == bust.

So what was plan three?  Well, dump over NFS using native commands, of course.  Luckily the system administrator was well versed in NFS and the commands we were about to use.  So there was not a lot of explaining, I didn’t have to type commands, and my only real job (as it should be in this type of situation) was mindful and guided keystroke monitoring.  Here is what we did.

• To start off, please review this great guide on setting up NFS via the Ubuntu community wiki.  It does not dive deeply into NFS.  It just gives the basics on how to get it up and running on a live system or Live CD.  Of course, mileage may vary depending on distro, so beware.  Once it was all set up we navigated to the correct directories and prepared to acquire.  When all was done the system we were acquiring had a remote share mounted to /tmp/target.
• Before we started copying files we needed to keep track of everything we did.  Of course we could have done that as soon as we logged into the system, but as it was the system administrator performing the actions and this was not necessarily a “Live Acquisition” it was not completely necessary and we needed to have a good place, like an NFS mounted directory, to copy our files.  So we changed to the /tmp/target directory and ran the script command to record our actions and sent it to an appropriately titled file.
• Next we needed to record the system time.  Although we did note the BIOS time when the system was booting up, we need to record the time displayed by the operating system.  Running a quick date command proved enough.  No redirection was necessary as we were already recording everything into a file.
• We needed to know what type of system we were running our commands on.  A quick uname accomplished this.  Of course different distros also have different methods of providing version information about themselves.  Look to website for the distro you are working with to determine which file to check in and then cat that information so that it is displayed to the screen and thereby written to your script file.
• In order to get all of the system information written to the correct directory we needed several commands.  Unfortunately, if the system had been compromised and a rootkit had been deployed, there was the potential that critical information would not be copied.  To help alleviate some concern behind this issue we decided to run a few commands.  First we needed to know where each of our commands were located.  This was determined through a simple which command.  This let us know where the system thought the program was located and allowed us to call it directly as we did not want to rely on the $PATH environment variable.  Next we needed to get the versions for each command so that we knew which versions of the commands we were working with in case of any problems or other issues such as non-standard programs.  Most commands will take either a –version or -V and provide specific details about the command.  In this case we needed to know the version of md5sum, dd, and split that was provided by the system.  Once this information was displayed we had it recorded.  Next we took the md5sum of each of these commands just to be sure.  If necessary these outputs could be checked against known good versions from known good sources of the distro.  If really paranoid and the tools are available, other hashing tools could be used.  Although all of these tools could be “rooted” it is not very likely.  Plus, in this instance, we were not too worried about a rootkit, but you never know.
• Now were were ready to copy the information to our mounted directory.  The hard drive was recognized as hda by the operating system.  This means that we needed to copy /dev/hda using the dd command.  But since we were copying a large hard drive (80 GB +) we needed to use the split command to help with errors.  Now, I did not know this before we started, but I heard from one of my co-workers and it was confirmed by the system administrator, that copying large files (everything is a file in *nix) to a NFS mounted directory was not completely reliable.  To help control this I was told that copying to 2GB files would be the best way to handle it.  If something failed then it would be easier to pick up from the last good file and just calculate the proper offsets to start the copying process from  that point instead of the beginning.  Here is the command we ended up running: /bin/dd if=/dev/hda | /usr/bin/split -d -b 2000m - /tmp/target/server_name.dd. Although you can review the split man page to understand the full command, I will point out two things.  The dash “-” is necessary because it tells the command to use the STDIN from the pipe as the input file.  The “.” at the end of the command is also necessary because it is what split uses to number the files it creates.
• Depending on the system this will run for quite a while.  But before you run off for coffee and donuts or to analyze another system you’ll want to run one more command.  That command is the date command.  To do this just type date and press enter.  This will automatically run the date command once the previous command has completed running.  This time and date gets recorded into the script and the full run time is recorded for use in reporting and future efforts.

Although all of this took some effort to do, the end result was that I had acquired the system in a manner that I could use multiple tools to evaluate the information.  Once this was all complete I moved onto the second webserver where we had similar issues with acquiring the hard drive as the system was running on the same blade server.  I would love to tell you that I just followed my previous steps and that all went well.  Unfortunately I cannot.  We did perform all of the steps but as soon as we hit enter on the dd command the system virtually locked up.  This came as quite a surprise as the previous system did not take a significant performance hit during the acquisition.  We stopped the command and did quick review of the logs on this new system.  The log was full of I/O errors and we could only assume that the hard drive was slowly failing.  Therefore, plan three == fail.  That is where the system administrator came to my rescue.  As the system did not have any problem backing up every night we decided to just tar the operating system over to the NFS mounted directory.  Although this would not get everything off of the drive, it would at least get some of the information and some is better than none, at least at this point. Here how the tar was accomplished.

• Follow all of the previous steps to get the information about the system and the commands.
• Next you need to understand the version of tar provided by the distro you are working with.  Read the man page just to be sure you have the options determined correctly.  Because of the way that tar interacts with NFS (thank you, schism) it is probably going to be necessary to use the “–one-file-system” option.  This means that you will have to acquire each of your mounted directories separately.  For instance, if you have one partition mounted at /, one at /boot, and one at /home the “–one-file-system” command will not follow down the mounted directories.  You will have to do each separately.  The following command worked like a charm and then repeated for each mount point.  /bin/tar –one-file-system -cvf /tmp/target/server_name_root.tar /

So, after four tries I had all of the information that I needed to start my analysis.  Well, at least for these systems.  This simple acquisition really threw me for a loop initially.  Not only did I have to pull out multiple acquisition techniques it also affected my time on site.  That might not seem like much, but these extra step meant that I was on site for an additionally two days.  No small impact on the customer.  But, it could have been a lot worse.  If I had not been prepared with fall back alternatives, then I could have spent even more time on site.  Not only would this have impacted the customer’s pocket book but it would have also sewn the seeds of doubt into my capabilities.  This could have, in turn, jeopardized their view of the results of the analysis as well as the potential for the continuation of a forensic, or any other, business relationship.

Go forth and do good things,

Don C. Weber

Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.