Security Ripcord

One of the keys to incident response is to reduce the gap between compromise and when an organization starts taking action.  There are a number of tricks to identify compromised hosts.  Most organizations retain all types of logs for any number of reasons.  Unfortunately, auditing and never really using logs for anything except for records retention can cause organizations to treat them as merely objects to move around and not necessarily utilize for any action.  Even organizations who perform monitoring and effective records retention have a tendency to store and forget about older logs.

Access to old logs becomes painfully necessary during an incident response.  Without a well thought out and tested access plan, review of older logs can be a tedious and time consuming effort.  This can be further complicated by resources such as server load caused by searching and network connectivity caused by transfer.  I have experienced a few of these problems recently and I thought I would share them so that you can use them to prepare to reduce the response gap within your organization.

Firewall logs get large quickly.   The size of the organization is usually going to determine the length of time logs will/can be retained.  How long does it take you to search all of your firewalls back six months?  This becomes very important when firewall management and log storage have been outsourced to another organization.  What are your service level agreements for request response times?  Have you tested this response time?

Firewall log review will help you identifying internal IP addresses that connection to external IP addresses.  Now what do you do?  Well, if your environment is made up of static IP addresses this works just fine.  What happens if your organization utilizes DHCP for connectivity?  How long do you retain logging information for your DHCP leases?  How fast does it take to correlate firewall search results with the DHCP leases?  If you cannot answer these questions then additional testing will be necessary.

Monitoring for specific IP addresses is one thing, but often the information you have may be related to specific domains.  This means that DNS request logs become very important.  All of the same review questions apply.  Concerns about outsourced DNS apply.  Testing applies.

Time for a little math.  Let’s say you have an IP address of a known malicious server on the Internet.  If it takes you two days to search your firewall logs for connections and then it takes you an additional two days to correlate that information with DHCP logs, your incident response gap is four days.  That is four days before you can even begin applying your incident response process.  Four days before you can start requesting legal permission to adhere to law and regulations where the system resides.  Does it take two days for legal approval?  Now you are up to six days.  Do you have data analysis personnel that can respond to all of your locations?  Add two more days to mail systems and data analysis doesn’t begin for eight days.

Hopefully this helps.  Please test your computer incident response plan.  When you do, think outside of the box.  Ask additional questions.  Request information and time how long it takes.  Conduct lessons learned followed up with specific and managed goals.

Go forth and do good things,

Don C. Weber

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.