I am not going to delve into too much about the topics covered in the class.  It is outlined for you on SANS’ website and, well, Rob and his crew worked very hard on pulling all of the concepts together.  For that you should attended the course or purchase the course material if you would like a deeper understanding.  However, there are a bunch of priceless illustrations that help the students understand some of the complex topics that can be confusing when first exposed to the information.  The ones that popped out to me the most were the images covering the Forensic Investigation Methodology, Date/Time correlations, and the Filesystem/Sleuthkit Review.  Each of them, at a glance provides excellent clarification.

So, to alleviate the concerns of Harlan and Rob, I learned a lot by attending the course.  I guess I am just one of those people.  I will admit that I was not as challenged as when I attended some of the other SANS trainings I have attended.  Certainly it was a fire hose for most of the attendees.    I just figure that this means that for the past two years I have been approaching incident response properly.  Rob’s class most validated the processes I have formed surrounding my acquisition and initial analysis techniques.    A lot of this came to me via Harlan’s training both professionally, individually, and via reading his blog and books.  Some of these concepts were also developed via the experiences of Chris Pogue.  His Sniper Forensics talk is a direct representation of many of the concepts I employ.  (It was good to finally meet him after two years.)

Since I am not going to go too deep into the concepts covered by the class (although they should shape some of the future content here) I will provide you with some of the notable quotes that came from Rob.

“…training the new breed of incident responder.”

Absolutely, SEC508 provides a sound foundation.  It exposes incident responders to the basics of the field.  Starting with a sound foundation is what is necessary.   (Tangent Alert!)  It also takes incident response and digital forensics out of the court room and back into the data center.  Which is important because the data center changes much faster than the court room.  By letting the court room lead our incident response processes we are limiting our capabilities to adapt to new threats and attack methodologies.  Let the court room keep up with us.

“…EMTs do not worry about adjusting evidence …”

Another statement enforcing the point I just made.  Of course, what should be noted is that EMTs approach an incident with a specific methodology.  They have a plan and they execute it.  When necessary, they deviate from that plan.  But familiarization and continuous training around the basics of that plan make it second nature to them.  This means that their actions can be accounted for and justified when evidence is necessary.

“Evidence integrity goes to the weight of what the evidence can be used for….”

Basically, be more concerned about the actions you have taken to gather information.  Once again, following your plan, knowing the basics, and documenting deviations.  Just because there is or is not a hash does not mean that, if necessary, the information will not be admissible during a court case.  But court cases should not be your major concern.  Consistent and repeatable process should be your concern.  This is necessary in case there is a need to repeat the data analysis in a court room, for a Board of Directors, or for a team of auditors.

“Tools do not have to be validated.  The output, what was found, is more important than the tool that was used to interpret the data.”

This is one of the first concepts that Harlan explained to me when I started working with him.  Different tools display information better than other tools (which is why we have a wide variety of them).  But just because a tools presents the data in a certain way, or has been doing so for X number of years, does not mean it is doing so correctly.  Other methods may be necessary to validate tool output.  This concept holds true for a perl/python script that was written last night by a kid in Poughkeepsie, NY or a long standing data analysis tool such as EnCase or FTK.

The forensic industry “is not a fad.  Organizations are spinning up internal teams to handle incident response and investigations.”

This is nothing new but it is a great validation.  Rob is exposed to a wide range of people from many different operational backgrounds.  This statement is also supported by the explosion of process and tool development in the digital forensic and incident response field.

I will end with a personal favorite of mine.  The following quote validates a realization I recently came to while cleaning up after an incident response.  If you have a weak heart, and hold onto old concepts  dearly, you may want to skip the following quote.  (I am paraphrasing because I just realized I didn’t write it down.)

“How many passes does it take to destroy data so that  forensic analysis tools cannot recover it?  One, yes you are correct.”

Yes, you read that right.  Only one pass is necessary.  Wow, that will save a lot of time not to mention a lot of energy related to processor intensive multiple writes using random data.  I am not going to track down all of the links that support this statement.  Basically, once information has been overwritten it cannot be accessed by the tools we typcially deploy.  Even advanced tools can only guess at the former state of a bit.  The cool thing is that since there are multiple layers to the file systems, there is a chance that a tool or process did not correctly overwrite the information.  This is a key concept covered by SEC508.  And as incident responders we also realize that just because data was destroyed in one location that it is not stored in some other location.  Which is why our processes include involving an organization’s network, workstation, server, and application administrators as well as management.  These people will understand where residual data resides within the organization.

So, to wrap this up, I highly recommend SEC508 to new and experienced incident response and digital forensic professionals.  You are going to learn something you did not know.  You are going to make contacts that will be invaluable in the future.  And, if you obtain the GIAC certification, you are going to have a valuable certification in a growing and increasingly important field that is having global impact.

Go forth and do good things,

Don C. Weber

Please help ITB by providing your submissions or letting us know about your intent to submit via the ITB Call Box.  We are also looking for article recommendations which we will place in the ITB Research Box so that others have good ideas as to what will help the DF/IR Community.  Obviously, if you would like to contribute but do not know what to write about, check out the ITB Research Box for recommendations.

Go forth and do good things,

Don C. Weber

The following timeline information was pulled from a system.  A complete timeline was generated using system artifacts parsed using System Combo Timeline and several TLN-based Enscripts.  Each line is in the order display within the timeline.  no lines have been removed.  This was a fairly lucky situation as many times unrelated activity needs to be purged from the timeline to see this type of detail and understand specific activities.  Specific comments will follow the lines I want to highlight.

2009 11 04 08:59:42|SysEvent.Evt – EVT|COMP-SYS|S-1-5-18|Service Control Manager/7035;Info;LiveUpdate,start
2009 11 04 08:59:42|SysEvent.Evt – EVT|COMP-SYS|N/A|Service Control Manager/7036;Info;LiveUpdate,running
2009 11 04 09:00:06|AppEvent.Evt – EVT|COMP-SYS|N/A|Symantec AntiVirus/16;Info;Virus definitions are current.
2009 11 04 09:00:12|SysEvent.Evt – EVT|COMP-SYS|N/A|Service Control Manager/7036;Info;LiveUpdate,stopped

Very interesting that many of these events seem to occur during some type of AV updating or scanning activity.  Although really just an interesting side note, this does show that AV can only prevent what it is programmed to understand.  The following events do not fall into that category.

2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18|Change Time (ctime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18|Modified Time (mtime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18\d42cc0c3858a58db2db37658219e6400_dbcab107-5342-4908-9e95-0aba2546f18b|Accessed Time (atime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18\d42cc0c3858a58db2db37658219e6400_dbcab107-5342-4908-9e95-0aba2546f18b|Change Time (ctime)
2009 11 04 09:19:09|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\S-1-5-18\d42cc0c3858a58db2db37658219e6400_dbcab107-5342-4908-9e95-0aba2546f18b|Created Time (crtime)

These files appear to be associated with Microsoft Cryptographic Service Providers.  I am still researching what this actually implies.  Please post a comment if you can provide some insight.

2009 11 04 09:21:00|SysEvent.Evt – EVT|COMP-SYS|S-1-5-18|Service Control Manager/7035;Info;UpsGSx,start
2009 11 04 09:21:00|SysEvent.Evt – EVT|COMP-SYS|N/A|Service Control Manager/7036;Info;UpsGSx,running
2009 11 04 09:21:00|SysEvent.Evt – EVT|COMP-SYS|S-1-5-18|Service Control Manager/7035;Info;UpsGSx,stop

This service is a malicious backdoor that has been related to Hydraq. Notice that new services do get logged to Windows Event Logs but only if your systems are configured to log.  But why did this service start?

2009 11 04 09:21:00|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\Tasks|Change Time (ctime)
2009 11 04 09:21:00|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\Tasks|Modified Time (mtime)

Ah, a modification to the Scheduled Tasks folder.  This could be an indication that a job was scheduled to run.

2009 11 04 09:21:00|Registry Hive: software|COMP-SYS|0|$$$PROTO.HIV/Microsoft/SchedulingAgent
2009 11 04 09:21:00|Registry Hive: system|COMP-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_UPSGSX/0000
2009 11 04 09:21:00|Registry Hive: system|COMP-SYS|0|$$$PROTO.HIV/ControlSet002/Enum/Root/LEGACY_UPSGSX/0000

More activity associated with the UPSxxx backdoor service starting.

2009 11 04 09:21:00|Task Log|COMP-SYS|”At1.job” (a.exe)    Started 11/4/2009 5:21:00 PM|0
2009 11 04 09:21:00|Task Log|COMP-SYS|”At1.job” (a.exe)    Finished 11/4/2009 5:21:00 PM    Result: The task completed with an exit code of (0).|0

Well, now, what do we have here?  A scheduled task running a strange executable.  And, at the exact same time that the UPSxxx backdoor service started.  Nobody said that timeline information will be presented down to the millisecond.  Obviously this action occurred before some of the other events.  Had the conversion from local time to UTC been off this data might have been lost and not associated with these events.

2009 11 04 09:31:58|SysEvent.Evt – EVT|COMP-SYS|N/A|Tcpip/4226;Warn;

Once, Harlan Carvey recommended that I get a subscription to EventID.net.  Here is where that subscription pays off.  The description for Event ID 4226 reads “TCP/IP has reached the security limit imposed on the number of concurrent TCP connect attempts.”  Well, now, it seems that there is a large number of outgoing connection attempts.  Additional comments from other users of EventID.net provide indications that this type of activity can be linked to malware.  And to think, this was alerted in the Windows Event Logs no less.  Did I mention that these should be turned on?  Starting to seem very useful right about now.

2009 11 04 09:35:30|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\snmpapi.dll|Accessed Time (atime)
2009 11 04 09:35:30|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\inetmib1.dll|Accessed Time (atime)
2009 11 04 09:35:30|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\netstat.exe|Accessed Time (atime)
2009 11 04 09:36:53|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\wbem\wbemperf.dll|Accessed Time (atime)
2009 11 04 09:36:53|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\systeminfo.exe|Accessed Time (atime)
2009 11 04 09:36:55|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\wbem\stdprov.dll|Accessed Time (atime)
2009 11 04 09:41:17|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\tasklist.exe|Accessed Time (atime)
2009 11 04 09:43:27|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\WINDOWS\system32\nbtstat.exe|Accessed Time (atime)
2009 11 04 09:46:58|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\RECYCLER|Change Time (ctime)
2009 11 04 09:46:58|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\RECYCLER|Modified Time (mtime)

Okay, here is the hot and juicy stuff.  Wow, look at those Access times.  Access times on executables are indicative of the execution of these programs.  “Netstat”, “Systeminfo”, “tasklist”, and “nbtstat”.  All one after another? Very interesting.  The accessed DLLs also show us that other programs might have been run around this time.  SNMP?  Simple Network Management Protocol – a little network management anyone? WBEM? Web-Based Enterprise Management.  Definitely more reconnaissance behavior.

I would also like to point out that this activity appears to have occurred approximately ten minutes after the backdoor was placed on this system using the scheduled task.  So, following logic, this system was compromised from another system within the network.  The attacker completed the working on that first system (possibly similar reconnaissance efforts) and then connected to this system via the backdoor.  Why the backdoor?  Well, logging is turned on.  There is no network logon event.  This could very well mean that the user’s actual credentials have not been compromised, YET.  Or, this could mean the attacker did not want to generate a logon event, or it could mean that he required some functionality provided by the backdoor.  We are starting to read into this a little too much at this point.  But our initial thoughts, compromised from another system on the internal network and performing reconnaissance steps, are still sound.

2009 11 04 10:12:46|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034337.properties|Accessed Time (atime)
2009 11 04 10:12:46|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034337.properties|Change Time (ctime)
2009 11 04 10:12:46|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034337.properties|Modified Time (mtime)
2009 11 04 10:13:10|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\Administrator\Local Settings\Temp\vmware-user\vmware-3856-mks-user-476.log|Accessed Time (atime)
2009 11 04 10:13:10|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\C\Documents and Settings\Administrator\Local Settings\Temp\vmware-user\vmware-3856-mks-user-476.log|Change Time (ctime)
2009 11 04 10:13:35|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP33|Created Time (crtime)
2009 11 04 10:13:35|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP33\change.log.1|Created Time (crtime)
2009 11 04 10:13:37|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034288.dll|Change Time (ctime)
2009 11 04 10:13:37|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034288.dll|Created Time (crtime)
2009 11 04 10:13:37|EnCase File TLN|COMP-SYS|COMP-SYS\Disk Image\D\System Volume Information\_restore{01E266C2-86F5-40B2-9145-B4252FFF29C3}\RP34\A0034287.dll|Change Time (ctime)

Ah, the System Restore Point.  This simple occurrence could be very helpful indeed.  It is quite possible that the Upsxxx backdoor was captured and stored in this System Restore Point.  This is very useful information especially if the malware has rolled over to a new version.  If your security team is looking for a different version of the backdoor, systems compromised with the version captured in this restore point may still be beaconing or allowing access to your environment.  Time to dive into these System Restore Points and see what they yield.

Hopefully you have found this helpful.

Go forth and do good things,

Don C. Weber

I follow a very specific initial process when appoaching compromises on Microsoft Windows systems.  Keyword searches, timeline generation, and registry analysis top my list unless inidcators point me in a different direction.  Keyword searches on unallocated space and the systems pagefile did not provide any positive information related to the goals.   So I moved onto identifying the backdoor’s service to have a little more information to use during timeline analysis.  Easy enough with a quick review of the Service Registry keys.

System\ControlSet001\Services\RaSuPoo – Date Modified: 1/12/2010 11:05:40 AM
Description – ** NONE PROVIDED **
DisplayName – ** NONE PROVIDED **
ImagePath – %SystemRoot%\System32\svchost.exe -k netsvcs
Parameters – Date Modified: 1/12/2010 11:05:40 AM
ServiceDll – c:\windows\system32\rasmon.dll
Security – Date Modified: 1/12/2010 11:05:40 AM

Right off the bat I noticed a few things that were interesting.  Neither the “Description” nor the “DisplayName” key values were present.  Although they are listed here it is only to note that they were missing completely.  Now these things might be unique to this system but they are definitely indicators that can be used to identify other compromised systems with in the environment, possibly even if the backdoor is rolled over to a different version by the attackers.

Something else that is important to note is the matching “Last Write” for the primary registry key and the subkeys “Parameters” and “Security”.  When these times do not match I rely on the times provided by the “Security” key to determine the likely date and time that the system was compromised.  As these times matched for this service I was initially inclined to utilize these times as indications of the initial compromise.

Additional review of the services using intelligence from the Internet located other malicious services.  The following Registry information represents one of these malicious services.

System\ControlSet001\Services\AppMgmt – Date Modified: 1/12/2010 11:05:40 AM
Description – Provides software installation services such as Assign, Publish, and Remove.
DisplayName – Application Management
ImagePath – %SystemRoot%\System32\svchost.exe -k netsvcs
Parameters – Date Modified: 10/14/2009 01:25:37 AM
ServiceDll – C:\Documents and Settings\Administrator\AppMgmt.dll
ServiceDllUnloadOnStop – 00000001
Security – Date Modified: 4/4/2005 12:24:16 PM

There are several interesting things to note about this process.  First of all is the location of the AppMgmt.dll.  This location should flag this service as malicious during a manual review of running services as well as when this service is running in memory.  The “Last Write” time of the “Parameter” key is a good indication of the date and time the system was compromised by this backdoor.  After additional review this seems to be the earliest indication of compromise.

Continuing with timeline analysis produced an interesting finding from the event logs.

2010 01 12 11:05:40|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaShniI,stop

As you can see, the backdoor service with a different random key name than the current registry setting has stopped.  Now, this could be typical behavior associated with the functionality of the backdoor or it could be indications of another malware rollover to a new version.  However, now we have additional indicators for the timeline analysis.  A little fancy searching produced sixteen other instances of randomly named services that stopped and produced an Event Log entry.  Timeline analysis also produced several instances of systems restarting as represented by the “Task Scheduler Service” starting.

2010 01 11 11:40:09|Task Log|INF-SYS|”Task Scheduler Service”   Started at 1/11/2010 9:40:09 AM|0

Continued review provided an even more interesting finding.  Instances of a LEGACY registry key for each “stopped” service associated with the backdoor.

2009 10 29 05:48:46|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RAS0HWK/0000

Additional investigation of the System Registry Hive produced the following information.

System\ControlSet001\Enum\Root\LEGACY_RAS0HWK – Date Modified: 10/27/2009 5:18:46 AM
0000 – Date Modified: 10/27/2009 5:18:46 AM
DeviceDesc – RaS0HWk
Service – RaS0HWk

Then, after trimming the fat from timeline entries the following details emerged.  I have trimmed these a bit for brevity.

2009 10 27 05:18:59|Task Log|INF-SYS|”Task Scheduler Service”   Started at 10/27/2009 13:18:59 PM|0
2009 10 27 05:18:46|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RAS0HWK/0000
2009 10 27 05:19:17|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaS0HWk,stop
2009 10 30 09:10:55|Task Log|INF-SYS|”Task Scheduler Service”   Started at 10/30/2009 17:10:55 PM|0
2009 10 30 09:10:43|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RAS1DBH/0000
2009 10 30 09:11:19|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaS1DbH,stop
[snip]
2010 01 11 01:10:49|Task Log|INF-SYS|”Task Scheduler Service”   Started at 1/11/2010 9:10:09 AM|0
2010 01 11 01:10:44|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RASX4NS/0000
2010 01 11 01:10:01|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaSX4nS,stop
2010 01 12 01:55:02|Registry Hive: system|INF-SYS|0|$$$PROTO.HIV/ControlSet001/Enum/Root/LEGACY_RASHNII/0000
2010 01 12 01:55:18|SysEvent.Evt – EVT|INF-SYS|S-1-5-18|Service Control Manager/7035;Info;RaShniI,stop

Now, this backdoor behavior is significant because I have not found any reference to this restart behavior in any of the Hydraq malware analysis reports.  It also means that the Last Write times for the “RaSuPoo” service registry key is not indicative of the time that the AppMgmt rolled over to the new backdoor.  Whether the restart behavior was intentional or not this behavior has successfully obfuscated some critical system information that may not have been apparent had it not been for timeline analysis.

Additional analysis of timeline data using the LEGACY data also revealed another significant finding.  Malware reports related to Hydraq provide mention of a backdoor service with the name “Upsxxx” where [xxx] are random.  Timeline review revealed a LEGACY key for this service.

System\ControlSet001\Enum\Root\LEGACY_UPSYVM – Date Modified: 10/27/2009 4:07:09 AM
0000 – Date Modified: 10/27/2009 4:07:09 AM
DeviceDesc – UpsyVm
Service – UpsyVm

The significance of this registry key is not the fact that it appears to have been created before the earliest RaSxxxx service keys.  Rather it is the fact that there are no Registry entries pertaining to the initial Upsyvm service keys.  This means that the original service keys were deleted once this backdoor was rolled over to the new backdoor.  Another significant finding revealed during timeline analysis.

Now, none of these indicators provide details about the “smoking gun” – manual activity – that has taken place on this system.  However, understanding all of these activities provides a complete understanding of the series of events that could lead up to manual malicious activity on other systems.  Details of system artifacts and backdoor activity times will help identify other compromised systems and other activities.  This information also provides new methods that are being used to obfuscate data and hide critical details about events following a compromise.  Hopefully the information provided here helps you overcome some of these techniques.

Go forth and do good things,

Don C. Weber

Access to old logs becomes painfully necessary during an incident response.  Without a well thought out and tested access plan, review of older logs can be a tedious and time consuming effort.  This can be further complicated by resources such as server load caused by searching and network connectivity caused by transfer.  I have experienced a few of these problems recently and I thought I would share them so that you can use them to prepare to reduce the response gap within your organization.

Firewall logs get large quickly.   The size of the organization is usually going to determine the length of time logs will/can be retained.  How long does it take you to search all of your firewalls back six months?  This becomes very important when firewall management and log storage have been outsourced to another organization.  What are your service level agreements for request response times?  Have you tested this response time?

Firewall log review will help you identifying internal IP addresses that connection to external IP addresses.  Now what do you do?  Well, if your environment is made up of static IP addresses this works just fine.  What happens if your organization utilizes DHCP for connectivity?  How long do you retain logging information for your DHCP leases?  How fast does it take to correlate firewall search results with the DHCP leases?  If you cannot answer these questions then additional testing will be necessary.

Monitoring for specific IP addresses is one thing, but often the information you have may be related to specific domains.  This means that DNS request logs become very important.  All of the same review questions apply.  Concerns about outsourced DNS apply.  Testing applies.

Time for a little math.  Let’s say you have an IP address of a known malicious server on the Internet.  If it takes you two days to search your firewall logs for connections and then it takes you an additional two days to correlate that information with DHCP logs, your incident response gap is four days.  That is four days before you can even begin applying your incident response process.  Four days before you can start requesting legal permission to adhere to law and regulations where the system resides.  Does it take two days for legal approval?  Now you are up to six days.  Do you have data analysis personnel that can respond to all of your locations?  Add two more days to mail systems and data analysis doesn’t begin for eight days.

Hopefully this helps.  Please test your computer incident response plan.  When you do, think outside of the box.  Ask additional questions.  Request information and time how long it takes.  Conduct lessons learned followed up with specific and managed goals.

Go forth and do good things,

Don C. Weber

The syscombotln tool has been updated to fix several bugs and time/date issues.  I have also decided to stop being lazy and updated all of the internal modules and external scripts/tools associated with this tool to properly handle the TLN format as Harlan outlined. This includes the TLN.EnScript which is NOT included in the syscombotln tool.

New functionality includes parsing the Windows XP setupapi.log file.  I have included this functionality due to a little analysis trick pointed out to me by Jason Luttgens and Jon Gross of Mandiant.  Basically any time a Windows Help (chm) file is executed the information is logged in the user’s HTML Help (hh.dat) file.  This information can be used to specify some Initial Infection Vector information.  This information, in turn, may be augmented by driver-based information which, in Windows XP, is logged in the setupapi.log file.  I cannot provide specifics at this time, but I can tell you that you will know suspicious entries when you see them.  (If anybody has specific examples, please provide them in the comments.)  Although I have not had time to parse the hh.dat file, I have had time to parse the setupapi.log file.  The syscombotln module for this file is very basic but it should handle all files well (please let me know if you experience cases where it does not).  An added benefit of parsing this log file is that external USB storage device installation information will also be added to your timelines.  And if there are anti-forensic efforts recommending deleting this log, you know we want to review it’s information and add it to our timelines.

IronGeek Resource:

Just a quick note.  When researching the information in the last post I ran across this great resource by IronGeek.  Once again he has posted some amazing content.  Take a look at his “Forensically interesting spots in the Windows 7, Vista and XP file system and registry” resource.  You might have known about this, but it is new to me, so just in case you missed it as well.

Scripts and Tools:

I have decided to start uploading my scripts and tools when I generate an update.  To this end I have created the Scripts and Tools page which includes some Window Registry tools (including some older RegRipper plugins) and a few Enscripts.  Check this page often for updates and new scripts/tools.  Leave comments with comments, updates, requests.  To help with consistency, I have also started using Subversion to help me track development of all my projects.  Basically because I have been brow-beating (unsuccessfully) Harlan to do the same with his tools.  I have started keeping all my projects on an external USB drive (which I backup often).  To keep each project separate I use the following steps.

1. Copy folder to Projects directory.
2. Type “svnadmin create /media/<usb drive>/Dev/Projects/Repo/<project name>”
3. Type “svn import <project name> file:///media/<usb drive>/Dev/Projects/Repo/<project name> -m “Initial Import”
4. Move original project directory “mv <project name> <project name>_bk”
5. Check out repository “svn checkout file:///media/<usb drive>/Dev/Projects/Repo/<project name> <project name>”
6. Double check files are there and work with it a little while.
7. Delete _bk

This works across Linux systems and should work on Windows systems using something like tortoisesvn.  Hopefully you find that useful for your script and tool development.

I don’t want to say too much about Into The Boxes here as everything that needs to be said is already on the sight.  Harlan Carvey, my partner in this project, has already created a post to welcome you all to the e-magazine.  We are hoping that you will find time to contribute to the first edition that will be released in the first week of January 2010.  This project is only going to be as successful as the people who are willing to share their knowledge and experiences.  Harlan and I will contribute as much as possible but we are hoping that the security community steps up to the plate, puts the word out, and starts providing technical and managerial input that security professionals working in the digital forensics and incident response field can implement.

I hope that you will visit the Into The Boxes website.  I also hope that you will help us make this a great resource for the security community.

Go forth and do good things,

Don C. Weber

AccessData’s FTK Imager Lite 2.6.1 provides security professionals and system administrators with valuable capabilities during all phases of the incident response process.  Whether trying to identify compromised systems, determine containment, or confirm eradication this tool provides powerful capabilities that can be immediately employed with minimal impact to the live system being analyzed.

There are two objectives for this tutorial:

• Demonstrate how to use FTK Imager Lite to acquire information from a live system.
• Introduce security professionals and system administrators to some of the steps that can be taken to quickly gather actionable intelligence from live systems.

Please let me know if you have any comments or issues with this tutorial.  I am also interested in requests for future tutorials.

Go forth and do good things,

Don C. Weber

If you don’t have FTK Imager Lite in your arsenal you should reconsider why you have not obtained it yet.  This tool is one of the most important tools when it comes to rapidly obtaining system information.  Put this tool on a USB drive, attach it to the system you are investigating, and quickly grab system memory, the registry, system event logs, a directory listing including deleted files, and any other system file you think might help your efforts.

Now this just needs a technique to gather volatile system information such as a list of processes and network connections and you have everything you need for a full blown system investigation.  Oh, wait, system memory contains all that information.  EXCELLENT!!!!!  I guess you better start looking at those memory analysis tools again.

Go forth and do good things,

Don C. Weber

With a copy of the remote system’s memory and an understanding of F-Response’s impact on that system an analysis of the system’s memory can be performed. Advances in memory analysis are rapidly moving forward. A good and free tool for memory analysis is Mandiant’s Memoryze. Memoryze parses the bit-stream memory file and generates XML output associated with the contents of the memory. To review these XML files and present it as human-readable information Mandiant’s developers have released Audit Viewer. Although Memoryze can be run manually, it is much easier to utilize the functionality of Audit Viewer to Launch Memoryze to analyze the memory file.

Audit Viewer

By clicking the Launch Memoryze button the analyst is presented with the usual configuration functionality. The analyst will point the tool to the Memoryze executable, the memory file, and the information output directory where the XML files will written. The other configuration considerations pertain to the information that will be collected from the memory file.

• MemoryDD will acquire memory from the system Memoryze is run on or from a Physical Drive such as those provided through F-Response’s functionality.

• ProcessDD will collect information pertaining to the process name or identification number provided. This will actually pull copies of the process executable and drivers associated with the process and copy them to the output directory.

• Driver Walk List will parse through the memory and determine the drivers being used by the system.

• DriverDD will collect information pertaining to a single driver name or all drivers. This will actually pull copies of the driver from memory and create driver files in the output directory.

• Processes will collect information pertaining to a single process or all processes. Specific information about a process can be selected. These include:

  • Handles

  • Sections

  • Ports

  • Strings

  • Imports

  • Exports

  • Injected DLLs

• Hook Detection – actually, I have to admit that I don’t know much about hook analysis. This information is key for determining the affects of malware on a system. Memoryze can be configured to collect information pertaining to:

  • System Service Descriptor Table call table

  • System Service Descriptor Table functions

  • Interrupt Descriptor Table

  • IRP tables

Launch Memoryze

Once Memoryze has completed its analysis of the memory file Audit Viewer can analyze the XML output. After pointing Audit Viewer to the output information details about the information can be reviewed. The following image shows how network connections can be identified. In this example, Audit Viewer shows the the Telnet service has an established connection with a remote system.

Telnet with Established Connection

Of course, F-Response also has an established connection. Of course, as we know from earlier we should expect to see two network connections associated with the F-Response process. The presence of only one connection shown in the following image could be a result of Memoryze’s parsing method or the exact time that the memory image was generated.

F-Response with Established Connection

The following image shows F-Response is connected to the log file name f-response-ent.exe.log. This is a good indicator of how Memoryze and Audit Viewer can provide information about running processes that cannot be determined from simply understanding that a process or service is running on a system.

F-Response with open log file

The default analysis activity associated with each process parsed by using the the Process configuration is just to grab information about each process. Audit Viewer provides the ability to output the process executable and all for driver files associated with the selected process. These, in turn, can be analyzed individually using malware analysis or code review.

Acquire F-Response Process

Depending on the system and the process being acquired the acquisition process could take one to five minutes. The analyst can determine when the acquisition process is complete by monitoring the Audit Viewer command window that is started for the acquisition process. Once the acquisition is completed the command window will close.

Audit Viewer parsing process files

As mentioned, the acquired files will be written to the output directory for further analysis.

Acquired Files

Well, I think this is going to mark the end of this series of posts.  Hopefully you have a better understanding of some of the techniques and tools utilized when performing quick incident responses for yourselves or your customers.

If there is something else you would like to see or that you would like me to talk about, just drop a comment and I will look into it.  For now, however, I will be moving onto some of those scripts I was talking about and one or two other ideas that I have to help advance these techniques and tools.

Go forth and do good things,

Don C. Weber