Security Ripcord

One of the keys to incident response is to reduce the gap between compromise and when an organization starts taking action.  There are a number of tricks to identify compromised hosts.  Most organizations retain all types of logs for any number of reasons.  Unfortunately, auditing and never really using logs for anything except for records retention can cause organizations to treat them as merely objects to move around and not necessarily utilize for any action.  Even organizations who perform monitoring and effective records retention have a tendency to store and forget about older logs.

Access to old logs becomes painfully necessary during an incident response.  Without a well thought out and tested access plan, review of older logs can be a tedious and time consuming effort.  This can be further complicated by resources such as server load caused by searching and network connectivity caused by transfer.  I have experienced a few of these problems recently and I thought I would share them so that you can use them to prepare to reduce the response gap within your organization.

Firewall logs get large quickly.   The size of the organization is usually going to determine the length of time logs will/can be retained.  How long does it take you to search all of your firewalls back six months?  This becomes very important when firewall management and log storage have been outsourced to another organization.  What are your service level agreements for request response times?  Have you tested this response time?

Firewall log review will help you identifying internal IP addresses that connection to external IP addresses.  Now what do you do?  Well, if your environment is made up of static IP addresses this works just fine.  What happens if your organization utilizes DHCP for connectivity?  How long do you retain logging information for your DHCP leases?  How fast does it take to correlate firewall search results with the DHCP leases?  If you cannot answer these questions then additional testing will be necessary.

Monitoring for specific IP addresses is one thing, but often the information you have may be related to specific domains.  This means that DNS request logs become very important.  All of the same review questions apply.  Concerns about outsourced DNS apply.  Testing applies.

Time for a little math.  Let’s say you have an IP address of a known malicious server on the Internet.  If it takes you two days to search your firewall logs for connections and then it takes you an additional two days to correlate that information with DHCP logs, your incident response gap is four days.  That is four days before you can even begin applying your incident response process.  Four days before you can start requesting legal permission to adhere to law and regulations where the system resides.  Does it take two days for legal approval?  Now you are up to six days.  Do you have data analysis personnel that can respond to all of your locations?  Add two more days to mail systems and data analysis doesn’t begin for eight days.

Hopefully this helps.  Please test your computer incident response plan.  When you do, think outside of the box.  Ask additional questions.  Request information and time how long it takes.  Conduct lessons learned followed up with specific and managed goals.

Go forth and do good things,

Don C. Weber

Reviewing my daily blog hits I came across another interesting search that directed somebody here.  “what a firewall and ids can tell you about an incident” Although I sarcastically commented on my Interesting Search Keywords page “That you need to turn on more logging” I have to admit that I meant it.  Firewalls and Intrusion Detection Systems will only provide the analyst (be it a professional incident responder or a system/network administrator) with a limited amount of information.  The real meat of an intrusion is going to come from the coalition of data from disparate logging resources.

• Firewall logs, depending on how logging is configured, is going to tell you about the connection attempts in and out of the network.  Most likely you are going to see the traffic that failed.  If the firewall has proxy capabilities you will be provided with specifics about the proxied traffic, also most likely the stuff that failed.  Depending on where the firewall is located within your infrastructure will determine how much noise will be present within the logs.  Internet facing firewalls will have a lot of noise and will therefore probably have failed logging toned down.  Internal firewalls, however, should not see much failed traffic.  So anything that shows up could protentially be interesting.  Comparing these logs could also be affective in determining the origin of specific attacks, were they intitiated internally or externally.
• IDS logs could be helpful or noisy as well.  These really depend on how well the security or network administrator has maintained the IDS sensors and where they have been placed.  Having every signature and capability turned on will lead to quite a bit of false positives to sort through.  But too much tweaking could leave administrators oblivious to reconnaissance efforts.  IDSes are great for determining if attackers are using known attack vectors to gain a beachhead within an environment.  But, like anti-virus engines, they are only as good as their signatures no matter how well the community and commercial companies stay on top of it.  I’m willing to bet it is going to be a long time before we see signatures for Clickjacking (BTW, Jeremiah Grossman recommends this article).
• These points aside, having the logs from both of these devices provides you with the best information of all.  What is the normal operational state of communications within the environment? There are no if, ands, or buts about it.  If you do not know what can be considered normal there is no way to asertain what is abnormal.  Having good logs from these devices will enable you to pin-point a specific time period that reconnaissance or even an attack occurred.  This could help you narrow your window of other investigation items be it system logs, file assess or write times, or registry modifications.

I said it on Twitter the other day, ” Logs are interesting, logs are fun, logs should be done by EVERYONE…..get to logging!!!” and I meant it.  It is very hard to investigate an intrusion or incident if there are no logs to review.  Bringing together the firewall logs, IDS logs, application logs, operating system logs, and anything else there is provides the foundation from which to build the response.  Without them you will just be staring at each other wondering what to do next.  And I’ll tell you what to do if you don’t have any logs.  Contain the incident, investigate as much as you can, return to a good operational state, disclose if you have to disclose, and TURN ON LOGGING.

Go forth and do good things,

Don C. Weber

I have to concede to Chris on several points of his latest post. I do so because:

1. He definitely has more experience, than I, deploying a variety controls in a variety of environments of varying size.
2. He definitely has more experience, than I, speaking to the capabilities of these controls and providing comprehensive and understandable analogies and examples.
3. He definitely has more time, than I, to correlate and integrate, free and expensive, disparate and concise literature and case studies to fuel his analogies and examples and employ them in a variety of circumstances.

After all, it is what he does for a living. And he is very good at it. That is why he is listed in my blogroll and the majority of the blogroll’s associated with my daily information security firehose. Hell, it is why he can list articles in many hard and soft copy information security publications.

Me, on the other hand, I am a security professional wielding my experiences and knowledge to the best of my ability to provide my employers and customers with the same level of service Chris provides despite my limitations due to time in service. I use my experiences with technology, interactions, and introspection to form my conclusions and present them as the very best solution for the situation. I will personally guarantee the deployment of every one of my recommendations and the provide mitigation suggestions when it is, as we know it will be, circumvented, exploited, outdated, outclassed, obsolesced, ineffective, unmanageable, flappable, overly expensive, or just plain wrong. In other words, I am confident and I am willing to make mistakes because I can fix them and the majority of the time I will not make them again.

I truly think that this whole blogging interaction started because of my attempt to be flamboyant about the topic to draw attention to it. Unfortunately, as most gussied up topics do, the central point of the discussion was lost for a while. Luckily, in his last post, Chris brought it back around. Let me try to talk about my point in very plain English.

I have a problem with vendors who are developing products that provide many security controls on one system (not UTMs, I’m talking firewalls performing a combination of spam detection, anti-virus, protocol analysis, data evaluation, and whatever else the vendor thinks will sell) and selling it as the ultimate solution for the perimeter of a company’s infrastructure. I have a problem with these solutions because the technologies they are combining on one system are not simple applications. They are robust technologies with a lot of complexity and I am afraid that the vendors will not take the interoperability of these technologies into considerations before they push them to market. I would much rather recommend to my employers and customers that we limit the utilization of such technologies to select portions of the internal network where they can provide the most value with the least concern. I feel much better placing tried and true simple, relatively speaking, controls at the locations associated with high risk. I don’t have full proof examples. I don’t have case studies to back up my hypothesis. I have my feelings and opinions. And, actually, since I am not dealing with Fortune 500 CEOs, CTOs, CISOs, and patent producing PHD weilding end users, I don’t really need it. In the realm of the small, limited budget, network, my feelings and opinions have been, to this point, sufficient.

Next, I don’t think I have a problem with purchasing a UTM to provide a combination of spam detection, anti-virus, protocol analysis, data evaluation, and whatever else the vendor thinks will sell because I believe that UTM developers have taken the complex nature of these technologies into consideration. I was hoping that somebody I know would respond by telling my, and their, readers whether or not UTM solutions are better than the “all-in-one” firewall solution advertised in the DarkReading article, and why. If I had to guess, because of my aforementioned lack of UTM experience, I would think that UTMs separate the responsibilities in much the same manner as role-based control.

Can anybody answer this question for me? It is all I really wanted out of the whole conversation.

So, Chris, I lay my King down so that we may reset the board and start the next conversation fresh. I think you are correct when you say that I need to provide more clarifying evidence during my conversations. I will take it to heart as much as I can in my day to day security related duties. I’ll even attempt to do so in my blogging. But, as my blog is more for personal edification, education, and venting I have a feeling that a few misguided and ill-informed opinions will slip in from time to time.

Go forth and do good things,

Don C. Weber

UTM, security, complexity, Security Ripcord, Chris Hoff

All of this started when I decided to react to the Kelly Jackson Higgins’ DarkReading Firewalls Ready for Evolutionary Shift article. Fast on its heals came a follow-up in reaction to a comment about UTMs. In the follow-up, since I have little experience with UTMs, I though I would call on a few people to validate or correct my assumptions as to how UTMs operate. Chris Hoff, however, took this as a call to arms and decided to teach me a thing or two about blogging while dissecting my carefully crafted viewpoint. Apparently, empirical demonstration is the wave of the future to be practiced by all who have a point to convey. In keeping with Chris’ effort I will attempt to comply on the limited research budget allocated by the senior management of Cutaway Security.

First, we need to start with the question. Chris wants to know how I came to the conclusion that:

single-sourced solution adds “complexity” and leads to “increased risk?”

Let’s start with the definitions from Webster.com.

complexity -
1 : something complex
2 : the quality or state of being complex

That won’t do.

complex -
1 a: composed of two or more parts : composite b (1)of a word : having a
bound form as one or more of its immediate constituents <unmanly is a
complex word> (2)of a sentence : consisting of a main clause and one or
more subordinate clauses
2: hard to separate, analyze, or solve
3: of, concerned with, being, or containing complex numbers <a complex
root> <complex analysis>

That’s better. My favorite is “2: hard to separate, analyze, or solve.” It is my favorite because it is the resultant situation when you combine things, especially complex things like…….oh, say…..firewalls, proxies, protocol analyzers, anti-spam engines, intrusion detection/protection capabilities, and SSL decryptors. In my mind, we are starting with things that are already complex and now we are combining them and thereby making them even more complex. It would be one thing if they merely worked side by side, but by configuring these technologies to interrelate we are not adding to the complexity we are multiplying the complexity. Certainly we already have some of these technologies interrelating, such as firewalls and proxies, and we have been doing this for a while. In fact, it has been manageable. But tack on a third, forth, fifth complex technology and I do not think that even the most intuitive management interface is going to be sufficient.

“Knock, knock….Don…..empirical, remember?” Oh, sorry, Chris.

To prove all of my points I thought about throwing up a firewall and installing several of these technologies and providing all of you with a break down of the complex installation and configuration procedures. I had also planned on researching each of the technologies for exploitable vulnerabilities and presenting the combined exploitable vulnerabilities as a complex series of doors to the internal data goldmine of any organization. But then I added the total funds allocated by the senior management of Cutaway Security and the project funding supplied by Chris and realized that I had already exceeded the budget by merely thinking about it. Oh, well. I guess some related vulnerabilities posted to SecurityFocus will have to suffice.

• Snort/Sourcefire DCE/RPC Packet Reassembly Stack Buffer Overflow Vulnerability
• Squid Proxy NTLM Authentication Buffer Overflow Vulnerability
• SpamAssassin Vpopmail and Paranoid Switches Remote Command Execution Vulnerability
• PCRE Regular Expression Library Multiple Integer and Buffer Overflow Vulnerabilities

Wait a cotton picking minute! All of these vulnerabilities only give the attacker privileges at the level that the application is running. No administrator or vendor would run these as root. And there are NEVER any privilege escalation vulnerabilities that would……opps…..hang on a sec….

• Linux Kernel Ptrace Local Privilege Escalation Vulnerability
• Linux Kernel CIFS Local Privilege Escalation Vulnerability

So, if you take the number of technologies implemented on a device, and you add them together you get a number higher than what you started with. And, if you count the number of risks associated with those technologies and you add them together (although I think you should multiply) you get a higher number of risks than you started with. All in a technology that is suppose to REDUCE RISK.

Although this might not have been the direction that Chris wanted me to go, this will have to suffice. By combining these technologies we are definitely increasing our risk in a way that the reduction in cost does not compensate. I guess the quickest example of this would have been to remind Chris that we ask our system administrators to limit the number of applications they place on a system. We do so to reduce the complexity of that resource, separate the vulnerabilities, and limit the amount of risk to the data traversing these applications. How can we, in good concience, request and even require them to follow these rules and then blatantly ignore them ourselves?

UPDATE:

In true blogging form (meaning I found it after I finished writing the original part of this post) I see that Mike Rothman has also weighed in on my opinion about complexity and risk. This post, in combination with Chris’, makes me realize that neither of the read the UTM Remix post very carefully. In the post I state that:

I wanted to cover this because UTM is actually a different animal then what I was originally addressing. Although I do not have any experience with Unified Threat Management, as a blogger I don’t feel ashamed jumping into it. I am sure that Chris Hoff, Rich Mogull, Lori MacVittie, Andy Willingham, or Alan Shimel will correct me if I am misguided.

Although Chris and Mike have labeled me a UTM hater, I have actually distanced UTM from my arguement as I believe they are different. I was hoping that the individuals listed could weigh in and clarify for me.

So, once again, I’ll state that I do not believe, albeit there is a lack of direct experience, that UTM fails to take the issue of increased risk due to complexity into consideration. Rather I believe, but need clarification on, that UTMs separate the functionality so that the data being analyzed is done so in a location that does not affect the overall operation of the system. Meaning, that if one of the technologies is exploited the attacker will not gain control of the entire system or will s/he be able to escalate their privileges in a manner that would adversely affect the rest of the system and the environment in which it has been deployed.

However, I do believe, and once again need clarification, that UTMs are expensive and difficult to deploy properly. Tarek has made a good comment about this to the UTM remix post. He points out that if the company has the money to spend on it, and he has seen instances where they do, that multiple UTMs from different vendors will help reduce some of the risk introduced by complexity. As it appears that he has worked in these environments more that I have, I will have to take his advice into consideration in the future.

Hopefully, all of this dispels Mike and Chris’ fear that I am spreading dis-information and FUD. And, yes, we are all still friends. But they do own me a beer.

Go forth and do good things,

Don C. Weber

Tarek, firewall, IDS, proxy, snort, squid, spamassassin, vulnerability, risk, Security Ripcord, Chris Hoff

I had a good comment from Tarek on the original Quit Complicating Our Controls post.

In fact, firewall were made to protect the different network segments or zones from each other by controlling who is supposed to talk to who using which protocol or application.

But later one, applications such as FTP, SIP, etc. started to open dynamic ports, and firewalls were forced to evolve and become more application aware. On the other hand Proxy Firewalls such as MS ISA – I know it’s a piece of crap – but such firewalls were able to see the application layer, add rules to prevent people from downloading ZIP and MP3 files, Inspect SMTP for spam, etc. So firewall vendors were forced to compete with them in this, especially that such Proxy Firewalls are popular in SOHO and SMB networks. And I think this is when UTM came to life. Vendors also competed with each other and each vendor wanted to have more features in his data-sheet, and I think were are going to see vendors announcing that their firewalls are the first to market with built in Coffee Makers.

I can agree with what you wrote here sometimes. For an ISP’s Data Centre or a Large Multinational Company this can be true. Having an all in one box is not the best choice. But when it comes to normal mid-range enterprises they can have a UTM, and in such case having two layers of clustered UTM’s from different vendors can protect them when complexity lead to a vulnerability in one box.

I wanted to cover this because UTM is actually a different animal then what I was originally addressing. Although I do not have any experience with Unified Threat Management, as a blogger I don’t feel ashamed jumping into it. I am sure that Chris Hoff, Rich Mogull, Lori MacVittie, Andy Willingham, or Alan Shimel will correct me if I am misguided.

Application firewalls have their own unique places. True, they definitely should not be lumped onto the controls you are using to separate your environments. But, the applications firewalls serve a purpose of, to use the terms loosely, deep-packet inspection and correlation directly associated with that specific application. These controls should be deployed directly in front of the application or application farm so that it can provide the most protection.

Now, as I mentioned UTMs are a different animal. They are taking the controls we are talking about separating and, although placing them on one device, keeping them separate. I imagine that when deployed correctly no one component of the system has the administrative access to the complete system. UTMs should be deployed so that while working the controls in parallel they are also passing the information off to controls that operate within their role with lesser privileges than the central system. So, technically, in the spirit of my original arguement, UTMs are acting correctly. Although we do get back to the whole, single point of failure issue, but that can be addressed by high availability.

Tarek, I think the real trap you are falling into is expanding the cost of your security controls by recommending separate UTMs within the same environment. Now, I do agree that having two would help reduce some risk, but not enough to offset the cost of the system, its installation, the training of employees, documentation of configuration, and many other things involved with deploying a solution. I image that deploying a UTM is in and of itself a very complicate task and organizations will have their hands full implementing one. Adding a second would just be cruel. Actually, by making this recommendation, you may be burning your bridges with your management. Remember, your management is going to be evaluating risk to reward and cost as well. If you are making recommendations that SIGNIFICANTLY increase cost and complexity without reducing risk SIGNIFICANTLY, you are running the risk of your management labeling your security group as a liability rather than an asset. Although this may not be true to your organization, I would think twice before making the dual UTM recommendation.

Go forth and do good things,
Don C. Weber

UTM, firewalls, Tarek, Security Ripcord, application firewalls

After reading LonerVamp’s take on the application aware firewall, I started to wonder why people constantly want to consolidate their controls. This is not a new debate and DarkReading’s article Firewalls Ready for Evolutionary Shift is not ground breaking as the integration of firewalls and other security technologies has been bouncing around for years. Indeed, here we see Marcus J. Ranum talking about it on “Date: Fri, 29 Mar 2002 12:00:29 -0500″:

I suspect you are referring to “intrusion prevention” – which is a hot new marketing term but basically everything that’s being billed
as “intrusion prevention” is just firewalling + antivirus with a bit of fresh paint on it.

I’m willing to bet he has changed his tune a little bit since then but the evolution of firewalls with additional integrated controls has been going on since 2002 at least.

Of course I can see why people desire to integrate the technologies.

• It is more cost effective to have two or more technologies on one piece of hardware.
• You only have to manage one box.
• The controls can augment each other more effectively and efficiently (according to the advertising on the box).
• Firewalls usually represent a choke point to external and potentially hostile environments.
• Vendors can market it as the Silver Bullet (no relation to Gary McGraw’s podcast) of controls.
• “The next-generation firewall will have greater blocking and visibility into types of protocols,” says Greg Young, research vice president for Gartner.
• etc

Well, I have a problem with all of this. Why are we making our controls more complex? Complexity leads to vulnerabilities. Vulnerabilities lead to exploits. Exploits lead to compromises. Compromises lead to loss.

Certainly, everything has vulnerabilities. But that is my problem with placing multiple controls on one system. Fine, if my firewall has a vulnerability then it can be bypassed and my organization is screwed until we can respond. But I would prefer that my firewall was not bypassed because of a vulnerability in another control like a protocol analyzer or an intrusion detection system. Oh wait, these will be newer technologies with better software development practices so there should not be any additional vulnerabilities that allow for exploitation of the system or bypass of the controls……RIGHT!!!!

Don’t get me wrong. I am all for developing new technologies that will allow organizations to analyze their traffic so that they get a better picture of what is traversing and exiting their networks. I just think they will be more effective if they are deployed so that they augment each other’s control measures instead of threatening them by increasing the risk through complexity. Controls should reduce risk, not increase it.

So, when considering how to protect your data please do not cut corners. Evaluate your data distribution and dissemination, consider your architecture, determine which controls will increase efficiency while increasing security, and then deploy those controls so that they augment each other effectively.

Go forth and do good things,
Cutaway

LonerVamp, firewalls, IDS, controls, security, DarkReading, Security Ripcord