Security Ripcord

There are several reasons that I am drawn to IT security and incident response.

• The discovery of what occurred.
• Protecting a business and its employees from people doing them harm.
• The need for a breadth and depth of knowledge in many areas.

When I was but a young security professional I always wanted to actively go head to head with a malicious hacker.  Last month, I finally got my chance.

Tom Liston (OMG, he has a Wikipedia page!?!) and myself, of InGuardians, came to our client’s site and integrated ourselves with the other security team members already on-site.  Our addition rounded out the team with a good cross-section of skill sets and experiences.  Perfect for what we were going to experience in the next few days.

By the time Tom and I arrived on-site the incident response team understood that the client’s website was being modified to serve malware to its visitors but they did know know how the modifications were being accomplished.  They had already start getting resources in place to provide data that would get us answers our questions.

• Snapshots of the dynamic websites were being saved to a strategic location.
• Web logs were configured to log centrally instead of locally.
• Full network packet captures were configured and stored in another location the incident responders could easily access.
• A method for remotely conducting data analysis and data acquisition was implemented.

Although I won’t say that I always ask for these things during an incident response, I will say that these are usually some of the recommendations that are made at the end of an engagement.  The reason these technologies are valuable is because they provide responders with the system and network-based artifacts that can be used to make quick decisions.

The response effort started normally.  We used our time to become familiar with the environment, located compromised webpages, and tried to piece together the initial compromise vector, persistence mechanisms, and propagation techniques being utilized by the attacker.  During this time we were able to devise methods for quickly locating anomalous activity within the information provided by the weblogs, packet captures, and file differences.  Analysis of systems did not detect anything significant so the team believed that the compromise was limited to the modification of dynamic web pages.

We quickly came to realize that the attackers had been modifying files for quite a while.  The specific code that provided them their initial access was eluding us and we made various recommendations to reduce the number of blatantly vulnerable web applications.  Our analysis identified various obfuscated PHP web administration tools (such as the web shell WSO and Web-based file manager webadmin.php) that the attackers were using to modify files and stomp the file’s metadata.  The combination of all of these things made timeline analysis very difficult and less fruitful than I am use to during active incidents.

This type of activity is fairly normal during incident response and we were starting to get to the point where our team was just going through the motions of detecting modifications and addressing them as quickly as possible.  While this was happening the systems and website administrators worked on individual sites and servers to close vulnerabilities (no small task, indeed).  These websites are the primary business of this client and each of the affected web applications and their servers had to be left online during the attack.  In other words, it was cost effective for them to keep us chasing the bad guys and to keep the sites alive despite the attack.

And then, we had contact with the enemy.

Tom and I had just sat down for our last day on-site.  All of our team had their systems up and running and we were going through our usual motions.  I had just sat down with my coffee when the client’s security manager walked in and told us that one of their primary sites was being blocked due to malicious code.  The only thing I can relate this moment to is somebody yelling “Contact Left!!” and the Marine squad turning in unison and entering the fray.  Each one of our team turned to our computers and start gathering information with a flurry of keystrokes.  One man starting rolling through the web logs for the last hour.  Another man starting pulling down the last hour’s worth of packet captures.  Tom located the malicious code in the infected webpage and I, using that information, tracked down the source of the inserted code to a PHP include file (obviously used since it would get tacked onto every page served by the web application).  I quickly handed the PHP code off to Tom for de-obfuscation (damn, he is seriously fast at it, too) and I turned to investigating the last “known good” snapshot of the website to try and get a good timeline for the team members conducting log analysis. At the same time, two of us remotely acquired images of the web servers in an attempt to detect any malicious interactions with the operating system and provide more detail to our timeline information.

All of our quick action payed off.  About fifteen minutes into this activity the team member analyzing the packet captures said “Holy sh*t!!! I just grabbed the latest packet capture and I think he is on the system right now.”  He paused and then said, “Holy sh*t!!  He’s got root.”  It turns out that all of our work during the week had hampered the attacker enough that s/he wanted to know what as going on with the system.  So, instead of merely managing the malicious activities via the web shell s/he dropped to interacting with the operating system.  The packet captures pointed us to several system-based artifacts which we grabbed for analysis.  This was a boon because until this point, as I mentioned, we had not detected this type of interaction (those web shells provide some nasty functionality – “It’s a feature!!”).  Now we knew the types of tools s/he was using, how s/he got them onto the system, how s/he was hiding them, and the tool’s capabilities.

About thirty minutes in the fray the excitement wore off.  We could see where the attacker logged off of the system (possibly moving onto their next compromised site).  We spent the rest of the day working through the information we found and firming up our findings and recommendations.  I actually flew out of there later that day; high on adrenaline and wishing I could stay.  After all, I would much rather stay and face the enemy than to return to the rear and write a report about it.

So, the incident response-related lessons learned from this engagement:

• Centralized logging of application logs and full network captures are critical during an incident response;
• Being prepared with remote acquisition and data analysis techniques is also extremely important;
• Team organization, tool familiarization, and overall experience will produce positive results from stressful situations;
• Management support and confidence breeds success.

These are the tools and concepts that will help your incident response team fight-the-fight while the mission critical assets stay online and ensures that everybody (client and consultant) gets paid.  When an incident response team asks for these resources or recommends they be put in place, they are not doing it because the technologies are cool.  They are making the recommendations because they are effective and will reduce the time and effort associated with the incident.  Pre-deploying these technologies or generating procedures for doing so will make them even more effective.

Go forth and do good things,

Don C. Weber

There, I said it.  If you are developing an Incident Response Plan, Business Continuity Plan, Disaster Recovery Plan, or any other important plan for that matter, you should consider utilizing some type of development life cycle.  In the title I refer to SDLC which could stand for Software or System Development Life Cycle depending on who you are.  But for this let’s go with System Development Life Cycle (SDLC).

Now I am not going to map this all out for you.  There are plenty of resources out there and, frankly, I am just too tired right now.  But I will tell you that each of these aforementioned plans require that your organization defines your requirements, designs a solution, develops the solution, implements what you have, tests everything, and then maintains the plan.  Of course you do not have to follow the traditional waterfall method as I have just described, but it is definitely a good place to start.  Actually, plans such as these would probably be better fitted by some type of group development strategy such as SCRUM.  This will allow you to identify the key personnel (Subject Matter Experts), managers (stake holders), and end-users (and anybody else that can provide positive input) and use them to define the requirements for success and then allow the team to determine how to best achieve the stated requirements.

Where did all of this come from?  Well, Hurricane Ike is in the Gulf of Mexico and it was originally headed straight for Corpus Christi.  Once the possibility of landfall here in CC was announce the town exploded with activity, including my house.  Food, water, clean yard, clean garage, board up the windows.  All of these things became last minute necessities that took the better part of a day to accomplish nearly completely.  What I learned from this all is that you might have a good plan, or inherited a good plan, but if you do not continue with testing and maintenance then the plan is going to fail.  A couple of personnal examples:  water filtering resources ran out of water, a run on plywood and particle board made many people wait for empending shipments to arrive, plywood coverings and their fastening locations warped over time making them hard or impossible to utilize, and more.  Small potatoes to a business but what about server power, alternate sites (are the buildings even still there?), backup management, location of personnel and their families, etc.  When was the last time that you have tested all of these?  Are your critical assets still the same?  What happens when you are backing everything up and you realize you have a security incident D’oh, two plans for follow simultaniously!!  Do you have the resources for that?

Using an SDLC will help you manage these plans better and insure that when you do need them, they work.  Good luck.

Go forth and do good things,

Don C. Weber

(NOTE: Slightly updated from the original.  I was very tired when I originally wrote this and I just wanted to add a few more clarifying points and examples.)

Recently I wrote about personal safety being the response of the individual.  Well, after a power outage last night I realize that I am a little deficient in my business continuity procedures.  Here is a list of thing that I realized after the fact.

• We were out of D-cell batteries.  All but one of our flashlights were dead.  The one good thing was that I knew exactly where the flashlight were and they were accessible (which is a big feat with a 2.5 year old in the house).
• We only had one candle.  No batteries and no flashlights means that there is going to be a need for another light source.  Backup, backup lightsource as you might say.  A household should have several candles in containers that will not drip wax as they burn, possibly through the night.  Also, remember that heat rises so be careful where you locate these for long periods of time.  Check what is above the candle and make sure it is not flammable.
• We don’t have a cooler.  Now that I don’t drink beer as much as I use to I never missed the cooler.  With short power outages you don’t have to worry about the things in the fridge but the power was out for 10 hours last night.  With a cooler I would have been able to put some of the necessities on some ice.  Luckily we immediately identified that we should not open the fridge and it remained cold enough that we don’t have to throw anything away.
• We went to sleep without extra blankets.  Although the nights have recently been warm, the power outage was caused by strong winds as a cold front was blowing in.  By the time I woke up I was cold.  Although my wife and I are resilient our two children are another story and I should have paid closer attention to their needs.

I am sure that I could have found plenty of other things that I had forgotten but as it was already late we just took the children to bed.  One good thing that came out of the power outage is that I got ten hours of sleep.  Now when is the last time that I could say that.

There was one other thing that I did before going to bed.  I unplugged as many electronic items that I could easily get to in the dark.  You don’t know if the power is going to come back on normally or if it is going to surge.  Unplugging things will ensure that the equipment is not damaged and help limit the chance that a piece of equipment will start a fire.  The fire danger is most important during power outages that occur at night because, well, you are asleep.

So, how can you protect yourself?  Well a quick Google search on “home power outage checklist” is one way.  eHow’s list definately would have help me.  Of course the Upper Hastings Ranch Association’s list points out that you should not use candles and stick with flashlights.  It also points out that generators should be kept outside and not run indoors.  This is very important and may seem like a no-brainer but it definitelly happens.  Here is a good reference about the dangers of Carbon Monoxide from the Environmental Protection Agency.

Cutaway