Security Ripcord

I mentioned in Gathering Hard Drive Serial Number and Information that I wanted a method to automatically document a hard drive’s information in Linux. I did come across a solution called Automated Image and Restore (AIR) that provides a GUI front-end to DC3DD and DD. I haven’t tested this as I really would prefer a command line utility that provides this functionality (let us know your comments on your experiences with AIR). Also, I thought that I preferred DCFLDD. I actually did until I started developing this script and I discovered some “unwanted functionality” which I will cover at the end of this post.

With all of this in mind, the Forensic Acquisition Information and Drive Data Script (faidds) was born. FAIDDS (available at Google Code) leverages the Python subprocess methods to run parted, hdparm, and sdparm to gather information about a specified hard drive (Python subprocess runs other executables, so MIND your file integrity). Once this information is gathered the hard drive is acquired using either DC3DD (default) or DCFLDD. The start and end times are recorded. All of the acquisition information is written to a log file. The script also generates a hash file associated with the acquired image leveraging the acquisition tool’s hashing functionality.

faidds.py -h

faidds.py [-h] [-D] [-dcfldd] [-m hash0,hash1] [-c size in G (1024*1024*1024)] [-s \"serial number\"] -d \”drive location\”‘
-h: help’
-D: debug information’
-s: user specified serial number. Default is to find serial number in drive info.’
-d: device file to acquire’
-c: size to split file in G (1024*1024*1024)’
-dcfldd: use dcfldd (default: dc3dd)’
-m: list of hash algorithms to use. Comma separated with no spaces. (default: md5)’

Here is how I use the script.

First I locate the hard drive I want to acquire. For this example I will attach a USB thumb drive to my system simulating that I am attaching a hard drive to the system using a write-blocker connected with a USB cable. To detect the newly attached drive I run the “dmesg | tail” command:

cutaway> dmesg | tail
[526855.404692] scsi 21:0:0:0: Direct-Access     LEXAR    JUMPDRIVE SECURE 2000 PQ: 0 ANSI: 0 CCS
[526855.406731] sd 21:0:0:0: Attached scsi generic sg6 type 0
[526855.409391] sd 21:0:0:0: [sdf] 502880 512-byte logical blocks: (257 MB/245 MiB)
[526855.415142] sd 21:0:0:0: [sdf] Write Protect is off
[526855.415152] sd 21:0:0:0: [sdf] Mode Sense: 43 00 00 00
[526855.415159] sd 21:0:0:0: [sdf] Assuming drive cache: write through
[526855.418722] sd 21:0:0:0: [sdf] Assuming drive cache: write through
[526855.418737]  sdf: sdf1
[526855.471513] sd 21:0:0:0: [sdf] Assuming drive cache: write through
[526855.471520] sd 21:0:0:0: [sdf] Attached SCSI disk
cutaway>

As you can see, I attached a 256 MB LEXAR USB thumb drive and it was assigned the device file “sdf”. Next, I move into the directory where I want to acquire the drive image. This could be the local hard drive but more than likely it will be some other media such as an external USB hard drive or a shared drive. To acquire this drive I decided to have it output the MD5 and SHA1 hashes of the drive I am acquiring.

cutaway> sudo python ../faidds/trunk/faidds.py -D -m md5,sha1 -d /dev/sdf
[sudo] password for cutaway:
Enter YES to acquire/dev/sdf: YES
/sbin/parted /dev/sdf print
/sbin/hdparm -I /dev/sdf
/usr/bin/sdparm –inquiry /dev/sdf
_ �Nk
System Time Zone Is: CDT
Start Time: March 18 2012 17:49:36 UTC
Acquisition command: /usr/bin/dc3dd log=./_ �Nk_20120318174936_hash.txt rec=on if=/dev/sdf hash=md5 hash=sha1 of=./_ �Nk_20120318174936.dd
[!!] opening log `./_\001 \205N\024\027k_20120318174936_hash.txt’: Invalid or incomplete multibyte or wide character
Try `/usr/bin/dc3dd –help’ for more information.
dc3dd aborted at 2012-03-18 12:49:36 -0500

Stop Time: March 18 2012 17:49:36 UTC
Traceback (most recent call last):
File “../faidds/trunk/faidds.py”, line 192, in <module>
ONF = open(onf,’w')
IOError: [Errno 84] Invalid or incomplete multibyte or wide character: ‘drive_data__dev_sdf__\x01 \x85N\x14\x17k_20120318174936.txt’
cutaway>

Lucky for me I selected this thumb drive for testing. Apparently, LEXAR did not set up the serial number properly (easily readable). In this case it appears that the device returns serial number information that is “multibyte or wide character.” This situation is not unique for this USB thumb drive and could occur when acquiring older USB and Flash drives. Thus the “user specified serial number” option was born.

cutaway> sudo python ../faidds/trunk/faidds.py -D -m md5,sha1 -d /dev/sdf -s testing
Enter YES to acquire/dev/sdf: YES
/sbin/parted /dev/sdf print
/sbin/hdparm -I /dev/sdf
/usr/bin/sdparm –inquiry /dev/sdf
System Time Zone Is: CDT
Start Time: March 18 2012 18:03:07 UTC
Acquisition command: /usr/bin/dc3dd log=./testing_20120318180307_hash.txt rec=on if=/dev/sdf hash=md5 hash=sha1 of=./testing_20120318180307.dd

dc3dd 7.0.0 started at 2012-03-18 13:03:07 -0500
compiled options:
command line: /usr/bin/dc3dd log=./testing_20120318180307_hash.txt rec=on if=/dev/sdf hash=md5 hash=sha1 of=./testing_20120318180307.dd
device size: 502880 sectors (probed)
sector size: 512 bytes (probed)
257474560 bytes (246 M) copied (100%), 30.2548 s, 8.1 M/s

input results for device `/dev/sdf’:
502880 sectors in
0 bad sectors replaced by zeros
eaa52cfb7b1b37d2b94b8d371e0e47a8 (md5)
602c713c00c9a05c8f0ec76f9c3f2f7581da7edd (sha1)

output results for file `./testing_20120318180307.dd’:
502880 sectors out

dc3dd completed at 2012-03-18 13:03:37 -0500

Stop Time: March 18 2012 18:03:37 UTC
cutaway> ls -al
total 251452
drwx—— 1 cutaway cutaway       328 2012-03-18 13:03 .
drwx—— 1 cutaway cutaway      4096 2012-03-18 12:18 ..
-rwx—— 1 cutaway cutaway      2529 2012-03-18 13:03 drive_data__dev_sdf_testing_20120318180337.txt
-rwx—— 1 cutaway cutaway 257474560 2012-03-18 13:03 testing_20120318180307.dd
-rwx—— 1 cutaway cutaway       643 2012-03-18 13:03 testing_20120318180307_hash.txt
cutaway>

DC3DD provides us with an excellent record of  our acquisition method and the hash values for the acquisition.

cutaway> cat testing_20120318180307_hash.txt

dc3dd 7.0.0 started at 2012-03-18 13:03:07 -0500
compiled options:
command line: /usr/bin/dc3dd log=./testing_20120318180307_hash.txt rec=on if=/dev/sdf hash=md5 hash=sha1 of=./testing_20120318180307.dd
device size: 502880 sectors (probed)
sector size: 512 bytes (probed)
257474560 bytes (246 M) copied (100%), 30.2548 s, 8.1 M/s

input results for device `/dev/sdf’:
502880 sectors in
0 bad sectors replaced by zeros
eaa52cfb7b1b37d2b94b8d371e0e47a8 (md5)
602c713c00c9a05c8f0ec76f9c3f2f7581da7edd (sha1)

output results for file `./testing_20120318180307.dd’:
502880 sectors out

dc3dd completed at 2012-03-18 13:03:37 -0500

cutaway>

Of course, if you prefer (as I thought I did) you could use DCFLDD. The command line options are the same with the exception that the user needs to include the “-dcfldd” option.

cutaway> sudo python ../faidds/trunk/faidds.py -D -m md5,sha1 -d /dev/sdf -s testing -dcfldd
Enter YES to acquire/dev/sdf: YES
/sbin/parted /dev/sdf print
/sbin/hdparm -I /dev/sdf
/usr/bin/sdparm –inquiry /dev/sdf
System Time Zone Is: CDT
Start Time: March 18 2012 18:07:00 UTC
Acquisition command: /usr/bin/dcfldd hashlog=./testing_20120318180700_hash.txt conv=noerror,sync if=/dev/sdf of=./testing_20120318180700.dd
7680 blocks (240Mb) written.
7857+1 records in
7858+0 records out
Stop Time: March 18 2012 18:07:31 UTC
cutaway>

You can run this command in the same directory as the other acquisition because the output files are all time stamped. The hashlog file for DCFLDD is not as robust as DC3DD, but don’t worry, I capture similar information in the drive acquisition data file.

cutaway> ls -al
total 502913
drwx—— 1 cutaway cutaway       328 2012-03-18 13:07 .
drwx—— 1 cutaway cutaway      4096 2012-03-18 12:18 ..
-rwx—— 1 cutaway cutaway      2529 2012-03-18 13:03 drive_data__dev_sdf_testing_20120318180337.txt
-rwx—— 1 cutaway cutaway      2519 2012-03-18 13:07 drive_data__dev_sdf_testing_20120318180731.txt
-rwx—— 1 cutaway cutaway 257474560 2012-03-18 13:03 testing_20120318180307.dd
-rwx—— 1 cutaway cutaway       643 2012-03-18 13:03 testing_20120318180307_hash.txt
-rwx—— 1 cutaway cutaway 257490944 2012-03-18 13:07 testing_20120318180700.dd
-rwx—— 1 cutaway cutaway        46 2012-03-18 13:07 testing_20120318180700_hash.txt
cutaway> cat testing_20120318180700_hash.txt
Total (md5): eaa52cfb7b1b37d2b94b8d371e0e47a8
cutaway> tail drive_data__dev_sdf_testing_20120318180731.txt
Integrity word not set (found 0x7c3c, expected 0x91a5)

/usr/bin/sdparm –inquiry /dev/sdf
/dev/sdf: LEXAR     JUMPDRIVE SECURE  2000

System Time Zone Is:CDT

Start Time: March 18 2012 18:07:00UTC
Acquisition command: /usr/bin/dcfldd hashlog=./testing_20120318180700_hash.txt conv=noerror,sync if=/dev/sdf of=./testing_20120318180700.dd
Stop Time: March 18 2012 18:07:31UTC
cutaway>

Okay, who sees what I see? Notice anything different? What about the MD5 sums recorded for the captured data (the files with the “.dd” extension)? They are the same: eaa52cfb7b1b37d2b94b8d371e0e47a8. Hmm. Okay, what about the file sizes? The image created by DC3DD is 257474560 and the image created by DCFLDD is 257490944. Interesting. Let’s run the MD5SUM command against these files.

cutaway> md5sum testing_20120318180307.dd testing_20120318180700.dd
eaa52cfb7b1b37d2b94b8d371e0e47a8  testing_20120318180307.dd
a3aa87600be5559117c7450b5475cd0c  testing_20120318180700.dd
cutaway>

Wait a minute!!!! That is not right. And it is not the same hash value recorded by DCFLDD during acquisition. What could be the issue here? Could the second image be corrupted? Let’s check their similarity using SSDEEP (space added in the following output to prevent the stupid emoticon).

cutaway> ssdeep testing_20120318180307.dd testing_20120318180700.dd
ssdeep,1.1–blocksize:hash:hash,filename
6291456:Io/uDcy8TiPJY+Mj7C6llllllllllllaG3KfJh8rliaS6H: DOcrTiPJY+Mj7CGKfJolNH,”/media/InG-Storage/Research/Dev/data_acquisition_test/testing_20120318180307.dd”
6291456:Io/uDcy8TiPJY+Mj7C6llllllllllllaG3KfJh8rliaS6: DOcrTiPJY+Mj7CGKfJolN,”/media/InG-Storage/Research/Dev/data_acquisition_test/testing_20120318180700.dd”
cutaway>

These files appear to be exactly the same except for something at the end of the file. What could that be?

Ahh, it does appear to be exactly the same except that there are 16,384 null bytes attached to the end of the file created by DCFLDD. Well, we could remove those if we REALLY needed to. I am guessing that this was caused by the fact that I used the “conv=noerror,sync” option to handle bad blocks during acquisition. To test I simply ran the command by itself without this option.

cutaway> sudo /usr/bin/dcfldd hashlog=./testing_20120318180700_hash_noerror.txt if=/dev/sdf of=./testing_20120318180700_noerror.dd
7680 blocks (240Mb) written.
7857+1 records in
7857+1 records out
cutaway> ls -al
total 754653
drwx—— 1 cutaway cutaway      4096 2012-03-18 13:40 .
drwx—— 1 cutaway cutaway      4096 2012-03-18 12:18 ..
-rwx—— 1 cutaway cutaway      2529 2012-03-18 13:03 drive_data__dev_sdf_testing_20120318180337.txt
-rwx—— 1 cutaway cutaway      2519 2012-03-18 13:07 drive_data__dev_sdf_testing_20120318180731.txt
-rwx—— 1 cutaway cutaway    300656 2012-03-18 13:27 oketa_image_diff.png
-rwx—— 1 cutaway cutaway 257474560 2012-03-18 13:03 testing_20120318180307.dd
-rwx—— 1 cutaway cutaway       643 2012-03-18 13:03 testing_20120318180307_hash.txt
-rwx—— 1 cutaway cutaway 257490944 2012-03-18 13:07 testing_20120318180700.dd
-rwx—— 1 cutaway cutaway        46 2012-03-18 13:40 testing_20120318180700_hash_noerror.txt
-rwx—— 1 cutaway cutaway        46 2012-03-18 13:07 testing_20120318180700_hash.txt
-rwx—— 1 cutaway cutaway 257474560 2012-03-18 13:40 testing_20120318180700_noerror.dd
cutaway> md5sum testing_20120318180307.dd testing_20120318180700.dd testing_20120318180700_noerror.dd
eaa52cfb7b1b37d2b94b8d371e0e47a8  testing_20120318180307.dd
a3aa87600be5559117c7450b5475cd0c  testing_20120318180700.dd
eaa52cfb7b1b37d2b94b8d371e0e47a8  testing_20120318180700_noerror.dd
cutaway>

Yes, that is it. DCFLDD adds extra bytes to account for the rest of the file out to the end of the last block. The options that tells DC3DD to manage bad blocks is the “rec=on” option. As I have this selected for the DC3DD command it appears that it does not add these additional null bytes to the end of the file.

In other words, “Know Your Tools.”

Go forth and do good things,

Don C. Weber

When acquiring data I am always worried about writing down a hard drive’s part number and serial number incorrectly. Sometimes the print is so small that an “8″ will look like a “B” or maybe the information has been obscured by other markings, stickers, or time. Some acquisition techniques such as HELIX Pro Version 3 and FTK Imager provide this information in their drive acquisition information file and I don’t need to worry about it. But HELIX Pro doesn’t really work for me any more and I find myself booting the system using a different Linux Live CD before falling back to HELIX, if necessary.

My favorite Live CD for data acquisition is Backtrack version 4 or 5 because it is also a part of my assessment toolkit. BT4/5 has a forensic boot mode which does not use the local drive’s SWAP partition and does not automatically mount any internal or external drives. Since it is actively maintain I can usually use it to boot any system as long as that system has a DVD drive. It would be nice to have a CD-ROM version of BT5 for systems that do not have DVD drives or the ability to boot to USB. (Yes, those types of systems are still out there.)

Unfortunately BT does not provide a scripted ability to gather information about an attached drive. Software that does provide this information does so (as far as I can tell) by using C code to query the kernel for a drive’s information. If I had time to sit down and write the code I would, but I would rather have a simple way to use system commands to provide me with this information. This is where “hdparm” and “sdparm” come into the picture. Both of these commands are designed to query a hard drive and provide various types of information.

The following is the logical process that I use to gather the information I need about an attached hard drive.

The “dmesg” command provides information about any drive connected to the system. I use this command to show me where the operating system is attaching a new drive I have plugged into the system via USB or Firewire (directly or via a write blocker). In the following case the operating system is allowing access to the Western Digital hard drive via the “/dev/sdb” device file. This means that the system is treating the drive as a SCSI device. It also means (generally) that the “/dev/sda” device has already been taken by another SCSI hard drive.

cutaway@smash:~$ dmesg | tail -17
[  383.584108] usb 1-2: new high speed USB device number 7 using ehci_hcd
[  383.719545] scsi4 : usb-storage 1-2:1.0
[  383.722070] input: Western Digital  External HDD     as /devices/pci0000:00/0000:00:1d.7/usb1/1-2/1-2:1.1/input/input12
[  383.722264] generic-usb 0003:1058:0705.0005: input,hidraw3: USB HID v1.10 Device [Western Digital  External HDD    ] on usb-0000:00:1d.7-2/input1
[  384.717116] scsi 4:0:0:0: Direct-Access     WD       3200BEV External 1.75 PQ: 0 ANSI: 0
[  384.736576] sd 4:0:0:0: Attached scsi generic sg2 type 0
[  384.738391] sd 4:0:0:0: [sdb] 625142448 512-byte logical blocks: (320 GB/298 GiB)
[  384.738961] sd 4:0:0:0: [sdb] Write Protect is off
[  384.738969] sd 4:0:0:0: [sdb] Mode Sense: 23 00 00 00
[  384.740263] sd 4:0:0:0: [sdb] No Caching mode page present
[  384.740271] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[  384.745318] sd 4:0:0:0: [sdb] No Caching mode page present
[  384.745326] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[  384.798801]  sdb: sdb1
[  384.801173] sd 4:0:0:0: [sdb] No Caching mode page present
[  384.801181] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[  384.801188] sd 4:0:0:0: [sdb] Attached SCSI disk

Notice that the “dmesg” program provides manufacturer name, part number, and disk size information for this newly attached drive. The information that is missing is the drive’s serial number. To gather serial number we will need another method.

Linux operating systems usually come with the “hdparm” program as a core utility. This program is designed to “get/set SATA/IDE device parameters” from hard drives attached to the system. The following is the information this program provides pertaining to a locally attached hard drive. By locally attached I mean that this hard drive is directly connected to the system’s mother board.

cutaway@smash:~$ sudo hdparm -i /dev/sda

/dev/sda:

Model=ST9250827AS, FwRev=3.AAA, SerialNo=5RG08HAJ
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
BuffType=unknown, BuffSize=8192kB, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=488397168
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio1 pio2 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=yes: unknown setting WriteCache=enabled
Drive conforms to: unknown:  ATA/ATAPI-4,5,6,7

* signifies the current active mode

As you can see, by using the “hdparm” command, we now know the hard drive’s manufacturer part and serial number. Using the “-I” option actually gives us more information which could be useful to document.

cutaway@smash:~$ sudo !!
sudo hdparm -I /dev/sda
[sudo] password for cutaway:

/dev/sda:

ATA device, with non-removable media
Model Number:       ST9250827AS
Serial Number:      5RG08HAJ
Firmware Revision:  3.AAA
Transport:          Serial
Standards:
Used: unknown (minor revision code 0×0029) cutaway@smash:~$ sudo sdparm -C capacity /dev/sdb
/dev/sdb: WD        3200BEV External  1.75
blocks: 625142448
block_length: 512
capacity_mib: 305245.3

Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical         max     current
cylinders       16383   16383
heads           16      16
sectors/track   63      63
–
CHS current addressable sectors:   16514064
LBA    user addressable sectors:  268435455
LBA48  user addressable sectors:  488397168
Logical  Sector size:                   512 bytes
Physical Sector size:                   512 bytes
device size with M = 1024*1024:      238475 MBytes
device size with M = 1000*1000:      250059 MBytes (250 GB)
cache/buffer size  = 8192 KBytes
Nominal Media Rotation Rate: 5400
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec’d by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16  Current = 16
Advanced power management level: 254
Recommended acoustic management value: 254, current value: 0
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
Enabled Supported:
*    SMART feature set
Security Mode feature set
*    Power Management feature set
*    Write cache
*    Look-ahead
Host Protected Area feature set
*    WRITE_BUFFER command
*    READ_BUFFER command
*    DOWNLOAD_MICROCODE
*    Advanced Power Management feature set
SET_MAX security extension
*    48-bit Address feature set
*    Device Configuration Overlay feature set
*    Mandatory FLUSH_CACHE
*    FLUSH_CACHE_EXT
*    SMART error logging
*    SMART self-test
*    64-bit World wide name
*    IDLE_IMMEDIATE with UNLOAD
Write-Read-Verify feature set
*    WRITE_UNCORRECTABLE_EXT command
*    {READ,WRITE}_DMA_EXT_GPL commands
*    Gen1 signaling speed (1.5Gb/s)
*    Gen2 signaling speed (3.0Gb/s)
*    Native Command Queueing (NCQ)
*    Phy event counters
*    Device-initiated interface power management
*    Software settings preservation
*    SMART Command Transport (SCT) feature set
Security:
Master password revision code = 65534
supported
not     enabled
not     locked
frozen
not     expired: security count
not     supported: enhanced erase
Logical Unit WWN Device Identifier: 5000c5000990cb81
NAA             : 5
IEEE OUI        : 000c50
Unique ID       : 00990cb81
Checksum: correct

It would be nice if this command also provided the manufacturer name for the drive as the “dmesg” command did, but it is not completely necessary. A quick Google search shows that the Model Number for this drive is related to a Seagate drive.

Unfortunately we run into a problem with the “hdparm” command. This program does not work for drives attached to a system’s USB or Firewire ports. I am not exactly sure why this occurs, but it does have to do with the fact that drives attached via USB or Firewire do not return drive information when the kernel makes a “HDIO_GET_IDENTITY” request. Here is an example.

cutaway@smash:~$ sudo hdparm -i /dev/sdb
/dev/sdb:
HDIO_GET_IDENTITY failed: Invalid argument

To gather similar information we need to use the “sdparm” command. This command allows us to “access SCSI modes pages; read VPD pages; send simple SCSI commands” for a SCSI-based hard drive. To get a hard drive’s serial number using “sdparm” we need to directly query the VPD “page” that contains this information. Sometimes the “–inquiry” command will provide this information and sometimes it will not. By querying the “sn” page directly we can insure we get the hard drive’s part number and serial number with a single command.

cutaway@smash:~$ sudo sdparm –page=sn  /dev/sdb
/dev/sdb: WD        3200BEV External  1.75
Unit serial number VPD page:
WD-WXE308NF5233

If we need information pertaining to the drive’s capacity we will need to run an actual SCSI command to query the data from the drive. For this we will use the “-C” option and provide it the parameter “capacity”.

cutaway@smash:~$ sudo sdparm -C capacity /dev/sdb
/dev/sdb: WD        3200BEV External  1.75
blocks: 625142448
block_length: 512
capacity_mib: 305245.3

Be aware that the information provided by this command is dependent on how the drive responds to information requests. Most main stream hard drives and manufacturers will return good data. Some manufacturers and drive types (such as Flash drives) may not return the information you expect or require. Therefore, your mileage may vary.

This is all well and good, but as you will find out the “sdparm” program is not installed as a core utility on most Linux distributions. It has to be specifically installed. For Live CDs you may or may not be provided with this utility and this holds true for Backtrack as well. At least it did. The latest release, Backtrack 5r2 does include the “sdparm” program for the specific purpose of gathering this information from both the hard drive being aquired and the destination hard drive.

I have considered writing a python script that gathers this information. My first attempt queried “HDIO_GET_IDENTITY” and worked for local drives but not external drives. Thus leading me down this rabbit hole. I guess I will have to leverage Python’s subprocess capabilities to use the “sdparm” command to gather this information reliably. I was holding off on this script until “sdparm” was included in BT5r2. Now that it is released I guess I need to get busy. Check back soon.

Go forth and do good things,

Don C. Weber

When I started working for IBM’s Emergency Response Team I was a little intimidated about walking into a client’s environment and quickly providing incident response leadership. Luckily I was trained by Chris Pogue and Harlan Carvey to consider three things when I got on-site:

• What are you trying to answer?
• What data do you need to answer it?
• How do you get that data?

Obviously the first question is the hardest to answer (although, perhaps, not the most challenging). I often find myself having Car Talk-style conversations where you talk about the situation for an hour or two and then somebody says something to the effect of: “Oh, yeah. And we have this service running over here.”

It always amazes me how similar many of the security issues are across companies with decent security programs and teams. The security teams work hard to identify situations before they happen but incidents still occur and the teams struggle with pulling all of the information together. Disciplined incident response comes with experience. Not just experience within the security team, but experience throughout the organization. Most companies have information technology (IT) divisions with silo-ed experience (I am not saying it is bad, just natural). The web application developers and administrators know the web applications and web logs. Database administrators know the database logs, how verbose they are, how to retrieve them, and what can be turned on and off. Network administrators know what is logging where, what has been disabled, what can be turned up. Etc, Etc. Of course there is some overlap, but this example demonstrates that a lot of internal interaction may be required to initiate an incident response.

This silo-ing can complicate things but it is often manageable. The situations that prove to be more challenging are those that involve external services and equipment. Security teams are not usually consulted when service level agreements with external entities are formulated. Any third-party organization worth their salt will be prepared to provide information and action during critical times. But experience is the only way to determine the proper channels and methods of request to initiate the actions necessary to facilitate an efficient incident response. Acquiring data, removing services, increasing logging, and physically accessing a facility are just a few of the things that could increase the gap between incident identification and response.

Another thing that comes into play during an active incident response is the review of unique data types. I have assisted in several instances where an incident involved a web application running on an Apache server. The Apache web logs did not include any POST information which may have provided artifacts relating to the incident. As these web logs were the primary method for detecting anomalies the security team quickly realized that with the default configuration they may have been missing a few things. Digging a little deeper by including web application administrators and developers additional logs were identified. In a recent case the development team used Apache log4j to produce excellent activity information for debugging. As it did not introduce a performance hit to the server the developers just left the debugging code alone. The end result was a detailed log of the web server activity. The only real hick-up was that each entry was undocumented and multi-lined making it difficult to review by common automated processes. But, in the end, very helpful.

So, again, we come back around to the same incident response-related conclusion: Preparation is KEY. Understanding how things interact and who is involved is critical to reducing the time involved during each step of an incident response. But where to start with preparation? The results of an actual incident or a penetration test provide excellent examples and supporting data. Incident response scenarios are also very helpful but can be difficult to design and get everybody involved. To be honest, I have been a big advocate of “incident response scenarios” and I have told many people to develop them. But, looking back, I realize how hard it is for an organization to do this. Heck, it was hard for me when I was specifically thinking about and devoting time to developing a good incident response scenario. Therefore I am going to change my future recommendations from  developing incident response scenarios to performing system walk-throughs.

I made this decision when I found myself leaning back in a chair, looking at a white-boarded diagram of a system (by this I mean a group of resources combined for a specific purpose), and trying to determine what we needed to understand what might have happened during an incident. It was excellent because I also found myself and the security team asking questions to which we did not have immediate answers. I also found myself thinking about how I would have accomplished a specific set of actions relating to the incident. Coming up with these possibilities lead me to asking additional questions and identifying other resources-of-concern and data that could be used to identify useful network and system artifacts.

Therefore, my recommendation is that security teams periodically perform a system walk-through for various IT implementations in their organization. See how systems walk-throughs work and let us know your experiences. Pick one thing at a time when you start and see where it takes you. Focus on the things that will provide you information about a specific activity. Brain-storm situations and talk about the system and network-based artifacts that would be helpful. Folllow up on these ideas and pull in personnel to provide additional information when necessary. This will increase their experience and begin the collaboration process that may prove vitally important in future incidents. I also believe that these interactions will help identify risks, weaknesses, and vulnerabilities that can be easily addressed and increase the overall security of the organization.

Go forth and do good things,

Don C. Weber

There are several reasons that I am drawn to IT security and incident response.

• The discovery of what occurred.
• Protecting a business and its employees from people doing them harm.
• The need for a breadth and depth of knowledge in many areas.

When I was but a young security professional I always wanted to actively go head to head with a malicious hacker.  Last month, I finally got my chance.

Tom Liston (OMG, he has a Wikipedia page!?!) and myself, of InGuardians, came to our client’s site and integrated ourselves with the other security team members already on-site.  Our addition rounded out the team with a good cross-section of skill sets and experiences.  Perfect for what we were going to experience in the next few days.

By the time Tom and I arrived on-site the incident response team understood that the client’s website was being modified to serve malware to its visitors but they did know know how the modifications were being accomplished.  They had already start getting resources in place to provide data that would get us answers our questions.

• Snapshots of the dynamic websites were being saved to a strategic location.
• Web logs were configured to log centrally instead of locally.
• Full network packet captures were configured and stored in another location the incident responders could easily access.
• A method for remotely conducting data analysis and data acquisition was implemented.

Although I won’t say that I always ask for these things during an incident response, I will say that these are usually some of the recommendations that are made at the end of an engagement.  The reason these technologies are valuable is because they provide responders with the system and network-based artifacts that can be used to make quick decisions.

The response effort started normally.  We used our time to become familiar with the environment, located compromised webpages, and tried to piece together the initial compromise vector, persistence mechanisms, and propagation techniques being utilized by the attacker.  During this time we were able to devise methods for quickly locating anomalous activity within the information provided by the weblogs, packet captures, and file differences.  Analysis of systems did not detect anything significant so the team believed that the compromise was limited to the modification of dynamic web pages.

We quickly came to realize that the attackers had been modifying files for quite a while.  The specific code that provided them their initial access was eluding us and we made various recommendations to reduce the number of blatantly vulnerable web applications.  Our analysis identified various obfuscated PHP web administration tools (such as the web shell WSO and Web-based file manager webadmin.php) that the attackers were using to modify files and stomp the file’s metadata.  The combination of all of these things made timeline analysis very difficult and less fruitful than I am use to during active incidents.

This type of activity is fairly normal during incident response and we were starting to get to the point where our team was just going through the motions of detecting modifications and addressing them as quickly as possible.  While this was happening the systems and website administrators worked on individual sites and servers to close vulnerabilities (no small task, indeed).  These websites are the primary business of this client and each of the affected web applications and their servers had to be left online during the attack.  In other words, it was cost effective for them to keep us chasing the bad guys and to keep the sites alive despite the attack.

And then, we had contact with the enemy.

Tom and I had just sat down for our last day on-site.  All of our team had their systems up and running and we were going through our usual motions.  I had just sat down with my coffee when the client’s security manager walked in and told us that one of their primary sites was being blocked due to malicious code.  The only thing I can relate this moment to is somebody yelling “Contact Left!!” and the Marine squad turning in unison and entering the fray.  Each one of our team turned to our computers and start gathering information with a flurry of keystrokes.  One man starting rolling through the web logs for the last hour.  Another man starting pulling down the last hour’s worth of packet captures.  Tom located the malicious code in the infected webpage and I, using that information, tracked down the source of the inserted code to a PHP include file (obviously used since it would get tacked onto every page served by the web application).  I quickly handed the PHP code off to Tom for de-obfuscation (damn, he is seriously fast at it, too) and I turned to investigating the last “known good” snapshot of the website to try and get a good timeline for the team members conducting log analysis. At the same time, two of us remotely acquired images of the web servers in an attempt to detect any malicious interactions with the operating system and provide more detail to our timeline information.

All of our quick action payed off.  About fifteen minutes into this activity the team member analyzing the packet captures said “Holy sh*t!!! I just grabbed the latest packet capture and I think he is on the system right now.”  He paused and then said, “Holy sh*t!!  He’s got root.”  It turns out that all of our work during the week had hampered the attacker enough that s/he wanted to know what as going on with the system.  So, instead of merely managing the malicious activities via the web shell s/he dropped to interacting with the operating system.  The packet captures pointed us to several system-based artifacts which we grabbed for analysis.  This was a boon because until this point, as I mentioned, we had not detected this type of interaction (those web shells provide some nasty functionality – “It’s a feature!!”).  Now we knew the types of tools s/he was using, how s/he got them onto the system, how s/he was hiding them, and the tool’s capabilities.

About thirty minutes in the fray the excitement wore off.  We could see where the attacker logged off of the system (possibly moving onto their next compromised site).  We spent the rest of the day working through the information we found and firming up our findings and recommendations.  I actually flew out of there later that day; high on adrenaline and wishing I could stay.  After all, I would much rather stay and face the enemy than to return to the rear and write a report about it.

So, the incident response-related lessons learned from this engagement:

• Centralized logging of application logs and full network captures are critical during an incident response;
• Being prepared with remote acquisition and data analysis techniques is also extremely important;
• Team organization, tool familiarization, and overall experience will produce positive results from stressful situations;
• Management support and confidence breeds success.

These are the tools and concepts that will help your incident response team fight-the-fight while the mission critical assets stay online and ensures that everybody (client and consultant) gets paid.  When an incident response team asks for these resources or recommends they be put in place, they are not doing it because the technologies are cool.  They are making the recommendations because they are effective and will reduce the time and effort associated with the incident.  Pre-deploying these technologies or generating procedures for doing so will make them even more effective.

Go forth and do good things,

Don C. Weber

I recently attended SANS Security 508 at SANS 2010-Orlando.  When I told Harlan Carvey that I was going to attend this training he was concerned that I would not be exposed to anything I had not already exposed myself to through work and personal effort.  When I arrived on-site I got the same feeling from Rob Lee although his concerned seemed to be more centered around the value added by the course to more experienced incident response professionals.  Well, although their concerns were valid, I have to say that attending this class was a very valuable experience from the networking I accomplished, to the new (to me) concepts about how file systems work, to the concerns about how some applications leverage that information to produce system artifacts.

I am not going to delve into too much about the topics covered in the class.  It is outlined for you on SANS’ website and, well, Rob and his crew worked very hard on pulling all of the concepts together.  For that you should attended the course or purchase the course material if you would like a deeper understanding.  However, there are a bunch of priceless illustrations that help the students understand some of the complex topics that can be confusing when first exposed to the information.  The ones that popped out to me the most were the images covering the Forensic Investigation Methodology, Date/Time correlations, and the Filesystem/Sleuthkit Review.  Each of them, at a glance provides excellent clarification.

So, to alleviate the concerns of Harlan and Rob, I learned a lot by attending the course.  I guess I am just one of those people.  I will admit that I was not as challenged as when I attended some of the other SANS trainings I have attended.  Certainly it was a fire hose for most of the attendees.    I just figure that this means that for the past two years I have been approaching incident response properly.  Rob’s class most validated the processes I have formed surrounding my acquisition and initial analysis techniques.    A lot of this came to me via Harlan’s training both professionally, individually, and via reading his blog and books.  Some of these concepts were also developed via the experiences of Chris Pogue.  His Sniper Forensics talk is a direct representation of many of the concepts I employ.  (It was good to finally meet him after two years.)

Since I am not going to go too deep into the concepts covered by the class (although they should shape some of the future content here) I will provide you with some of the notable quotes that came from Rob.

“…training the new breed of incident responder.”

Absolutely, SEC508 provides a sound foundation.  It exposes incident responders to the basics of the field.  Starting with a sound foundation is what is necessary.   (Tangent Alert!)  It also takes incident response and digital forensics out of the court room and back into the data center.  Which is important because the data center changes much faster than the court room.  By letting the court room lead our incident response processes we are limiting our capabilities to adapt to new threats and attack methodologies.  Let the court room keep up with us.

“…EMTs do not worry about adjusting evidence …”

Another statement enforcing the point I just made.  Of course, what should be noted is that EMTs approach an incident with a specific methodology.  They have a plan and they execute it.  When necessary, they deviate from that plan.  But familiarization and continuous training around the basics of that plan make it second nature to them.  This means that their actions can be accounted for and justified when evidence is necessary.

“Evidence integrity goes to the weight of what the evidence can be used for….”

Basically, be more concerned about the actions you have taken to gather information.  Once again, following your plan, knowing the basics, and documenting deviations.  Just because there is or is not a hash does not mean that, if necessary, the information will not be admissible during a court case.  But court cases should not be your major concern.  Consistent and repeatable process should be your concern.  This is necessary in case there is a need to repeat the data analysis in a court room, for a Board of Directors, or for a team of auditors.

“Tools do not have to be validated.  The output, what was found, is more important than the tool that was used to interpret the data.”

This is one of the first concepts that Harlan explained to me when I started working with him.  Different tools display information better than other tools (which is why we have a wide variety of them).  But just because a tools presents the data in a certain way, or has been doing so for X number of years, does not mean it is doing so correctly.  Other methods may be necessary to validate tool output.  This concept holds true for a perl/python script that was written last night by a kid in Poughkeepsie, NY or a long standing data analysis tool such as EnCase or FTK.

The forensic industry “is not a fad.  Organizations are spinning up internal teams to handle incident response and investigations.”

This is nothing new but it is a great validation.  Rob is exposed to a wide range of people from many different operational backgrounds.  This statement is also supported by the explosion of process and tool development in the digital forensic and incident response field.

I will end with a personal favorite of mine.  The following quote validates a realization I recently came to while cleaning up after an incident response.  If you have a weak heart, and hold onto old concepts  dearly, you may want to skip the following quote.  (I am paraphrasing because I just realized I didn’t write it down.)

“How many passes does it take to destroy data so that  forensic analysis tools cannot recover it?  One, yes you are correct.”

Yes, you read that right.  Only one pass is necessary.  Wow, that will save a lot of time not to mention a lot of energy related to processor intensive multiple writes using random data.  I am not going to track down all of the links that support this statement.  Basically, once information has been overwritten it cannot be accessed by the tools we typcially deploy.  Even advanced tools can only guess at the former state of a bit.  The cool thing is that since there are multiple layers to the file systems, there is a chance that a tool or process did not correctly overwrite the information.  This is a key concept covered by SEC508.  And as incident responders we also realize that just because data was destroyed in one location that it is not stored in some other location.  Which is why our processes include involving an organization’s network, workstation, server, and application administrators as well as management.  These people will understand where residual data resides within the organization.

So, to wrap this up, I highly recommend SEC508 to new and experienced incident response and digital forensic professionals.  You are going to learn something you did not know.  You are going to make contacts that will be invaluable in the future.  And, if you obtain the GIAC certification, you are going to have a valuable certification in a growing and increasingly important field that is having global impact.

Go forth and do good things,

Don C. Weber

The success of Into The Boxes Issue 0×0 was only possible because of the collaboration provided by members of the Digital Forensics and Incident Response community.  In order for this publication to continue we need more people to step up and provide their input.  As you can see from the first issue we are looking for input that can be implemented by people in the DF/IR fields.  This input can be in the form of detailed articles or quick tips.  All input will be given serious consideration.  The ITB editors will provide authors with recommendations to strengthen their write-ups to ensure the best value to the community and help the authors develop as DF/IR professionals and writers.

Please help ITB by providing your submissions or letting us know about your intent to submit via the ITB Call Box.  We are also looking for article recommendations which we will place in the ITB Research Box so that others have good ideas as to what will help the DF/IR Community.  Obviously, if you would like to contribute but do not know what to write about, check out the ITB Research Box for recommendations.

Go forth and do good things,

Don C. Weber

Most of the time the initial infection vector associated with APT-style attacks incorporate the client-side exploitation of vulnerabilities in any number of software.  Actually, when dealing with APT-style events I prefer “initial compromise vector” (ICV) as APT backdoors should not be considered or even referred to as malware because it provides an incorrect understanding to the whole incident response.  User and system activity coupled with the fact that many organizations do not detect a compromise for several months means that the ICV can be elusive.  It is much more likely that the actions following the initial compromise are going to persist over time.

The following timeline information was pulled from a system.  A complete timeline was generated using system artifacts parsed using System Combo Timeline and several TLN-based Enscripts.  Each line is in the order display within the timeline.  no lines have been removed.  This was a fairly lucky situation as many times unrelated activity needs to be purged from the timeline to see this type of detail and understand specific activities.  Specific comments will follow the lines I want to highlight.

2009 11 04 08:59:42|SysEvent.Evt – EVT|COMP-SYS|S-1-5-18|Service Control Manager/7035;Info;LiveUpdate,start
2009 11 04 08:59:42|SysEvent.Evt – EVT|COMP-SYS|N/A|Service Control Manager/7036;Info;LiveUpdate,running
2009 11 04 09:00:06|AppEvent.Evt – EVT|COMP-SYS|N/A|Symantec AntiVirus/16;Info;Virus definitions are current.
2009 11 04 09:00:12|SysEvent.Evt – EVT|COMP-SYS|N/A|Service Control Manager/7036;Info;LiveUpdate,stopped

Very interesting that many of these events seem to occur during some type of AV updating or scanning activity.  Although really just an interesting side note, this does show that AV can only prevent what it is programmed to understand.  The following events do not fall into that category.

2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18|Change Time (ctime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18|Modified Time (mtime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18\d42cc0c3858a58db2db37658219e6400_dbcab107-5342-4908-9e95-0aba2546f18b|Accessed Time (atime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18\d42cc0c3858a58db2db37658219e6400_dbcab107-5342-4908-9e95-0aba2546f18b|Change Time (ctime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18\d42cc0c3858a58db2db37658219e6400_dbcab107-5342-4908-9e95-0aba2546f18b|Created Time (crtime)

These files appear to be associated with Microsoft Cryptographic Service Providers.  I am still researching what this actually implies.  Please post a comment if you can provide some insight.

2009 11 04 09:21:00|SysEvent.Evt – EVT|COMP-SYS|S-1-5-18|Service Control Manager/7035;Info;UpsGSx,start
2009 11 04 09:21:00|SysEvent.Evt – EVT|COMP-SYS|N/A|Service Control Manager/7036;Info;UpsGSx,running
2009 11 04 09:21:00|SysEvent.Evt – EVT|COMP-SYS|S-1-5-18|Service Control Manager/7035;Info;UpsGSx,stop

This service is a malicious backdoor that has been related to Hydraq. Notice that new services do get logged to Windows Event Logs but only if your systems are configured to log.  But why did this service start?

2009 11 04 09:21:00|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\Tasks|Change Time (ctime)
2009 11 04 09:21:00|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\Tasks|Modified Time (mtime)

Ah, a modification to the Scheduled Tasks folder.  This could be an indication that a job was scheduled to run.

2009 11 04 09:21:00|Registry Hive: software|COMP-SYS|0|$$$PROTO.HIV/Microsoft/SchedulingAgent
2009 11 04 09:21:00|Registry Hive: system|COMP-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_UPSGSX/0000
2009 11 04 09:21:00|Registry Hive: system|COMP-SYS|0|$$$PROTO.HIV/ControlSet002/Enum/Root/LEGACY_UPSGSX/0000

More activity associated with the UPSxxx backdoor service starting.

2009 11 04 09:21:00|Task Log|COMP-SYS|”At1.job” (a.exe)    Started 11/4/2009 5:21:00 PM|0
2009 11 04 09:21:00|Task Log|COMP-SYS|”At1.job” (a.exe)    Finished 11/4/2009 5:21:00 PM    Result: The task completed with an exit code of (0).|0

Well, now, what do we have here?  A scheduled task running a strange executable.  And, at the exact same time that the UPSxxx backdoor service started.  Nobody said that timeline information will be presented down to the millisecond.  Obviously this action occurred before some of the other events.  Had the conversion from local time to UTC been off this data might have been lost and not associated with these events.

2009 11 04 09:31:58|SysEvent.Evt – EVT|COMP-SYS|N/A|Tcpip/4226;Warn;

Once, Harlan Carvey recommended that I get a subscription to EventID.net.  Here is where that subscription pays off.  The description for Event ID 4226 reads “TCP/IP has reached the security limit imposed on the number of concurrent TCP connect attempts.”  Well, now, it seems that there is a large number of outgoing connection attempts.  Additional comments from other users of EventID.net provide indications that this type of activity can be linked to malware.  And to think, this was alerted in the Windows Event Logs no less.  Did I mention that these should be turned on?  Starting to seem very useful right about now.

2009 11 04 09:35:30|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\snmpapi.dll|Accessed Time (atime)
2009 11 04 09:35:30|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\inetmib1.dll|Accessed Time (atime)
2009 11 04 09:35:30|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\netstat.exe|Accessed Time (atime)
2009 11 04 09:36:53|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\wbem\wbemperf.dll|Accessed Time (atime)
2009 11 04 09:36:53|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\systeminfo.exe|Accessed Time (atime)
2009 11 04 09:36:55|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\wbem\stdprov.dll|Accessed Time (atime)
2009 11 04 09:41:17|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\tasklist.exe|Accessed Time (atime)
2009 11 04 09:43:27|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\nbtstat.exe|Accessed Time (atime)
2009 11 04 09:46:58|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\RECYCLER|Change Time (ctime)
2009 11 04 09:46:58|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\RECYCLER|Modified Time (mtime)

Okay, here is the hot and juicy stuff.  Wow, look at those Access times.  Access times on executables are indicative of the execution of these programs.  “Netstat”, “Systeminfo”, “tasklist”, and “nbtstat”.  All one after another? Very interesting.  The accessed DLLs also show us that other programs might have been run around this time.  SNMP?  Simple Network Management Protocol – a little network management anyone? WBEM? Web-Based Enterprise Management.  Definitely more reconnaissance behavior.

I would also like to point out that this activity appears to have occurred approximately ten minutes after the backdoor was placed on this system using the scheduled task.  So, following logic, this system was compromised from another system within the network.  The attacker completed the working on that first system (possibly similar reconnaissance efforts) and then connected to this system via the backdoor.  Why the backdoor?  Well, logging is turned on.  There is no network logon event.  This could very well mean that the user’s actual credentials have not been compromised, YET.  Or, this could mean the attacker did not want to generate a logon event, or it could mean that he required some functionality provided by the backdoor.  We are starting to read into this a little too much at this point.  But our initial thoughts, compromised from another system on the internal network and performing reconnaissance steps, are still sound.

2009 11 04 10:12:46|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034337.properties|Accessed Time (atime)
2009 11 04 10:12:46|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034337.properties|Change Time (ctime)
2009 11 04 10:12:46|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034337.properties|Modified Time (mtime)
2009 11 04 10:13:10|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\Administrator\Local Settings\Temp\vmware-user\vmware-3856-mks-user-476.log|Accessed Time (atime)
2009 11 04 10:13:10|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\Administrator\Local Settings\Temp\vmware-user\vmware-3856-mks-user-476.log|Change Time (ctime)
2009 11 04 10:13:35|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP33|Created Time (crtime)
2009 11 04 10:13:35|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP33\change.log.1|Created Time (crtime)
2009 11 04 10:13:37|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034288.dll|Change Time (ctime)
2009 11 04 10:13:37|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034288.dll|Created Time (crtime)
2009 11 04 10:13:37|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034287.dll|Change Time (ctime)

Ah, the System Restore Point.  This simple occurrence could be very helpful indeed.  It is quite possible that the Upsxxx backdoor was captured and stored in this System Restore Point.  This is very useful information especially if the malware has rolled over to a new version.  If your security team is looking for a different version of the backdoor, systems compromised with the version captured in this restore point may still be beaconing or allowing access to your environment.  Time to dive into these System Restore Points and see what they yield.

Hopefully you have found this helpful.

Go forth and do good things,

Don C. Weber

The other day I was handed a system that was known to be compromised with Hydraq.  The goals were to determine when, how, and what happened after the compromise.  Locating the malicious process during memory analysis was easy with so many known system artifacts.  Not really very useful although it did determine that the rasmon.dll program did contain a domain name unique to the rest of the intelligence from the Internet.

I follow a very specific initial process when appoaching compromises on Microsoft Windows systems.  Keyword searches, timeline generation, and registry analysis top my list unless inidcators point me in a different direction.  Keyword searches on unallocated space and the systems pagefile did not provide any positive information related to the goals.   So I moved onto identifying the backdoor’s service to have a little more information to use during timeline analysis.  Easy enough with a quick review of the Service Registry keys.

System\ControlSet001\Services\RaSuPoo – Date Modified: 1/12/2010 11:05:40 AM
Description – ** NONE PROVIDED **
DisplayName – ** NONE PROVIDED **
ImagePath – %SystemRoot%\System32\svchost.exe -k netsvcs
Parameters – Date Modified: 1/12/2010 11:05:40 AM
ServiceDll – c:\windows\system32\rasmon.dll
Security – Date Modified: 1/12/2010 11:05:40 AM

Right off the bat I noticed a few things that were interesting.  Neither the “Description” nor the “DisplayName” key values were present.  Although they are listed here it is only to note that they were missing completely.  Now these things might be unique to this system but they are definitely indicators that can be used to identify other compromised systems with in the environment, possibly even if the backdoor is rolled over to a different version by the attackers.

Something else that is important to note is the matching “Last Write” for the primary registry key and the subkeys “Parameters” and “Security”.  When these times do not match I rely on the times provided by the “Security” key to determine the likely date and time that the system was compromised.  As these times matched for this service I was initially inclined to utilize these times as indications of the initial compromise.

Additional review of the services using intelligence from the Internet located other malicious services.  The following Registry information represents one of these malicious services.

System\ControlSet001\Services\AppMgmt – Date Modified: 1/12/2010 11:05:40 AM
Description – Provides software installation services such as Assign, Publish, and Remove.
DisplayName – Application Management
ImagePath – %SystemRoot%\System32\svchost.exe -k netsvcs
Parameters – Date Modified: 10/14/2009 01:25:37 AM
ServiceDll – C:\Documents and Settings\Administrator\AppMgmt.dll
ServiceDllUnloadOnStop – 00000001
Security – Date Modified: 4/4/2005 12:24:16 PM

There are several interesting things to note about this process.  First of all is the location of the AppMgmt.dll.  This location should flag this service as malicious during a manual review of running services as well as when this service is running in memory.  The “Last Write” time of the “Parameter” key is a good indication of the date and time the system was compromised by this backdoor.  After additional review this seems to be the earliest indication of compromise.

Continuing with timeline analysis produced an interesting finding from the event logs.

2010 01 12 11:05:40|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaShniI,stop

As you can see, the backdoor service with a different random key name than the current registry setting has stopped.  Now, this could be typical behavior associated with the functionality of the backdoor or it could be indications of another malware rollover to a new version.  However, now we have additional indicators for the timeline analysis.  A little fancy searching produced sixteen other instances of randomly named services that stopped and produced an Event Log entry.  Timeline analysis also produced several instances of systems restarting as represented by the “Task Scheduler Service” starting.

2010 01 11 11:40:09|Task Log|INF-SYS|”Task Scheduler Service”   Started at 1/11/2010 9:40:09 AM|0

Continued review provided an even more interesting finding.  Instances of a LEGACY registry key for each “stopped” service associated with the backdoor.

2009 10 29 05:48:46|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RAS0HWK/0000

Additional investigation of the System Registry Hive produced the following information.

System\ControlSet001\Enum\Root\LEGACY_RAS0HWK – Date Modified: 10/27/2009 5:18:46 AM
0000 – Date Modified: 10/27/2009 5:18:46 AM
DeviceDesc – RaS0HWk
Service – RaS0HWk

Then, after trimming the fat from timeline entries the following details emerged.  I have trimmed these a bit for brevity.

2009 10 27 05:18:59|Task Log|INF-SYS|”Task Scheduler Service”   Started at 10/27/2009 13:18:59 PM|0
2009 10 27 05:18:46|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RAS0HWK/0000
2009 10 27 05:19:17|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaS0HWk,stop
2009 10 30 09:10:55|Task Log|INF-SYS|”Task Scheduler Service”   Started at 10/30/2009 17:10:55 PM|0
2009 10 30 09:10:43|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RAS1DBH/0000
2009 10 30 09:11:19|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaS1DbH,stop
[snip]
2010 01 11 01:10:49|Task Log|INF-SYS|”Task Scheduler Service”   Started at 1/11/2010 9:10:09 AM|0
2010 01 11 01:10:44|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RASX4NS/0000
2010 01 11 01:10:01|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaSX4nS,stop
2010 01 12 01:55:02|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RASHNII/0000
2010 01 12 01:55:18|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaShniI,stop

Now, this backdoor behavior is significant because I have not found any reference to this restart behavior in any of the Hydraq malware analysis reports.  It also means that the Last Write times for the “RaSuPoo” service registry key is not indicative of the time that the AppMgmt rolled over to the new backdoor.  Whether the restart behavior was intentional or not this behavior has successfully obfuscated some critical system information that may not have been apparent had it not been for timeline analysis.

Additional analysis of timeline data using the LEGACY data also revealed another significant finding.  Malware reports related to Hydraq provide mention of a backdoor service with the name “Upsxxx” where [xxx] are random.  Timeline review revealed a LEGACY key for this service.

System\ControlSet001\Enum\Root\LEGACY_UPSYVM – Date Modified: 10/27/2009 4:07:09 AM
0000 – Date Modified: 10/27/2009 4:07:09 AM
DeviceDesc – UpsyVm
Service – UpsyVm

The significance of this registry key is not the fact that it appears to have been created before the earliest RaSxxxx service keys.  Rather it is the fact that there are no Registry entries pertaining to the initial Upsyvm service keys.  This means that the original service keys were deleted once this backdoor was rolled over to the new backdoor.  Another significant finding revealed during timeline analysis.

Now, none of these indicators provide details about the “smoking gun” – manual activity – that has taken place on this system.  However, understanding all of these activities provides a complete understanding of the series of events that could lead up to manual malicious activity on other systems.  Details of system artifacts and backdoor activity times will help identify other compromised systems and other activities.  This information also provides new methods that are being used to obfuscate data and hide critical details about events following a compromise.  Hopefully the information provided here helps you overcome some of these techniques.

Go forth and do good things,

Don C. Weber

System Combo Timeline has been updated.  If you use syscombotln you will want to get this new version as there is an important bug fix.  I have also updated regtln.pl and evtparse.pl to handle double-byte and non-printable characters better.  This helps when analyzing log and registry files from Windows systems with various language packages.  Of course I do so my just stripping out the unwanted characters from the data entries.  You do not loose this information as you always have the original files.  But if you want/need to review the original entries you will need to review the original files.  These two tools are also available individually on the Scripts and Tools page.

PrefetchFolderAnalysis2.EnScript is a new addition to the Scripts and Tools page.  This EnScript was originally developed by Kelcey Tietjen.  I have updated it so that it outputs in TLN format and writes a file to the Export directory configured in EnCase.  It uses the last run time to place the entry in the timeline at the last time the file was executed.  If you integrate the results of this EnScript with the system’s File Directory listing you will have this included with MAC times.  Special attention, however, should be paid to the data entry.  The data will contain the original path of the executed command and the number of times that it was executed.  Extremely helpful during analysis.

If you would like to integrate the output from PrefetchFolderAnalysis2.EnScript into your timelines generated from syscombotln just place the resulting file into the “output” directory with any other TLN formated files you have created.  Running the syscombotln script will concatenate them all together.

Go forth and do good things,

Don C. Weber

One of the keys to incident response is to reduce the gap between compromise and when an organization starts taking action.  There are a number of tricks to identify compromised hosts.  Most organizations retain all types of logs for any number of reasons.  Unfortunately, auditing and never really using logs for anything except for records retention can cause organizations to treat them as merely objects to move around and not necessarily utilize for any action.  Even organizations who perform monitoring and effective records retention have a tendency to store and forget about older logs.

Access to old logs becomes painfully necessary during an incident response.  Without a well thought out and tested access plan, review of older logs can be a tedious and time consuming effort.  This can be further complicated by resources such as server load caused by searching and network connectivity caused by transfer.  I have experienced a few of these problems recently and I thought I would share them so that you can use them to prepare to reduce the response gap within your organization.

Firewall logs get large quickly.   The size of the organization is usually going to determine the length of time logs will/can be retained.  How long does it take you to search all of your firewalls back six months?  This becomes very important when firewall management and log storage have been outsourced to another organization.  What are your service level agreements for request response times?  Have you tested this response time?

Firewall log review will help you identifying internal IP addresses that connection to external IP addresses.  Now what do you do?  Well, if your environment is made up of static IP addresses this works just fine.  What happens if your organization utilizes DHCP for connectivity?  How long do you retain logging information for your DHCP leases?  How fast does it take to correlate firewall search results with the DHCP leases?  If you cannot answer these questions then additional testing will be necessary.

Monitoring for specific IP addresses is one thing, but often the information you have may be related to specific domains.  This means that DNS request logs become very important.  All of the same review questions apply.  Concerns about outsourced DNS apply.  Testing applies.

Time for a little math.  Let’s say you have an IP address of a known malicious server on the Internet.  If it takes you two days to search your firewall logs for connections and then it takes you an additional two days to correlate that information with DHCP logs, your incident response gap is four days.  That is four days before you can even begin applying your incident response process.  Four days before you can start requesting legal permission to adhere to law and regulations where the system resides.  Does it take two days for legal approval?  Now you are up to six days.  Do you have data analysis personnel that can respond to all of your locations?  Add two more days to mail systems and data analysis doesn’t begin for eight days.

Hopefully this helps.  Please test your computer incident response plan.  When you do, think outside of the box.  Ask additional questions.  Request information and time how long it takes.  Conduct lessons learned followed up with specific and managed goals.

Go forth and do good things,

Don C. Weber