Security Ripcord

Following up on any project is key.  Talking to all involved about what has happened, why it happened, how it could have better, what worked very well, etc is the key to improvement.  So, why aren’t organizations that experience a security related incident able to prevent or curtail future incidents?  Basically because they are not following up the incident with a lessons learned, or they are not doing it properly.

Incident Response should be handled like any other project.  It should be managed.  As most incident responders are aware, SANS GCIH course outlines 6 distinct phases of an incident response:

1. Preparation
2. Identification
3. Containment
4. Eradication
5. Recovery, and
6. Lessons Learned

Ah, Lessons Learned is the final step.  Actually, it is circular, so the final step leads into the first step.   I say leads into whereas a better metaphor might be “feeds.”  Or in the terms of following flow charts, for you management types out there, the outputs of Lessons Learned are the inputs of Preparation.  Here’s a question for you though, during your preparation phase did you look at the methodologies that you use to improve how you identify, prioritize, address, and follow-up on your lessons learned?  Or, when you are finished with your lessons learned meeting, do you have a list of action items that have been assigned to a specific individual who understands the criteria for successfully completing the action?

I just asked two questions that took your simple lessons learned meeting from a quick five minute session to a thirty minute plus session.  Although you might think that the issues will drive how long this meeting will last, in actuality that is not correct.  With proper methodology, in other words a practiced plan, this meeting can still be very quick while producing the key outputs that are necessary to augment your preparation phase.

Silver Bullet time.  Yes, I know, you want the silver bullet that is going to help you increase the effectiveness of your lessons learned process.  Guess what!!  Most likely you already have it.  Do you use process management to plan your software, hardware, or infrastructure development?  Bingo, then you have the means to improve your lessons learned.  Start using the processes that you already have in place.  I say this because it is the fastest and cheapest method to gaining control over this process.

For those of you who do not have a process in place, never fear, I have one word for you: Why.  Just ask why.  But ask it five times.  Asking why five times is the technique for determining root cause.  Gasp, Root Cause Analysis.  It is used during the Six Sigma process as well as being integrated into other project development schemes.  In his 5 Whys post Eric Ries explains that Taiichi Ohno of Toyota fame wrote about this technique in his book Toyota Production System: Beyond Large-Scale Production.

Now, this might seem silly to some at first.  It will seem especially silly to those individuals and groups that are not use to management their projects or doing root cause analysis.  This is where the longer meetings are going to come into play.  Of course, this is true of any new process.  New things take time to understand and get use to performing.  For some people the learning curve on how to conduct themselves in these meetings is going to be a long and tough journey.  But by consistently applying one of these methodologies to your lessons learned process you will find that each of your meetings is shorter and more productive.  The implementors will be happier because they are being heard (if they are participating) and the managers and executives will be happier because of the increased productivity and effectiveness of the end results.

Certainly I have only touched on this topic briefly.  If you have techniques that have worked to improve the effectiveness of your lessons learned meetings, please share them with us in the comments.

Go forth and do good things,

Don C. Weber

http://www.sans.org/?utm_source=web&utm_medium=text-ad&utm_content=affiliate_link1&utm_campaign=Cutaway_Security

I was completely wrong about the outcome of the UFC® 94 ST-PIERRE VS PENN 2 fight.  I was hoping that BJ would pull it out but it was very apparent by half-way through the second round that GSP had the better strategy.

From an IT security stand point what can we learn from this?  Well, the first thing that comes to mind is the fact that this was the second fight between these two fighters.  The first fight was won by GSP in a split decision at the end of three rounds.  What this means to my point is that these fighters had experience with each other.  They had stood up and traded punches and also grappled on the ground.  This usually means that second fights are exciting because fighters change their tactics to overcome their opponents weaknesses while also strengthening their own.  In the work-up to the second fight GSP did just that.  BJ, however, did not.  He came out and fought the same fight, worked hard, did his best, and got beaten.

Can you see where I am going with this?  IT environments generally do not grow and change.  Their business and security controls fight the same good fight everyday but, generally, they remain stagnant or slowly build up extra layers to cover up, instead of fixing, weaknesses.  But the core usually remains the same and when push comes to shove the old weaknesses are exposed.

I would like to say that I have some cool way that would allow an environment to modify itself on a regular basis.  Some way to change or rotate paths, protocols, and technologies.  I do not.  It is not very cost effective to operate in this manner.  It takes time, money, knowledge, and man-hours to do it effectively.  Accuracy is also a factor as changes can lead to mistakes in configuration and implementation.  These factors impose a heavy burden on the profit margin which in turn means that the risks associated with more stagnant defense are acceptable.

Fortunately for us IT business and security is not just a one-on-one battle.  It might be at first, but as in real war (which IT resembles more than the UFC) once you get in a fight, if you can hold off your opponent for a sufficient amount of time, help will arrive.  This help can come in the form of either outside expertise or law enforcement.  The important part is to be in a position that you can identify when you are being attacked and hold off the opponent long enough to gain enough information about them, or the weaknesses they are trying to exploit, to make the appropriate adjustments and, if necessary, call in help.

GSP studied his opponent much like a malicious hacker will a target.  He analyzed his opponent  for exploitable weaknesses, trained on how to overcome those weaknesses, and then attacked them with overwhelming force.  Sound similar to Conficker/Downadup? BJ trained for the fight and I am sure did everything he and his staff could do to identify and train towards GSP’s weaknesses.  But what he was not prepared for was to quickly modify his strategy to protect his own weaknesses and give himself the ability to regain the initiative.  Thus, GSP was victorious.

So take a look around your business.  Is there anything that you could modify on a regular basis to maintain a more challenging target to your opponents?  Have you effectively trained your staff to calmly and effectively cope with the highly stressful events that surround an adverse event or malicious incident?  Have you looked at yourself from the outside to determine the weakness that are the most logical to attack?  Do you have a plan to protect internal assets when your defenses are breached?  Do you know who the calvary is and how to contact them in the case of an emergency?  Do they know who you are?

Take a lesson from BJ and GSP.  Realize that a stagnant target is more easily attacked and beaten by an intelligent and well-prepared opponent.  Remember, knowing you have a problem is the first step to a cure.  Also, listen to what other people say about your environment and approach to solutions, because you don’t know what you are doing wrong until somebody tells you.

By the way, great job BJ Penn and Georges St-Pierre.  Although GSP won you are both winners for training hard and giving it your all.  Keep up the good work.

Go forth and do good things,

Don C. Weber

Although we have seen some recent activity concerning a hacked road-side construction sign, you should be aware that this situation was documented on Jun 23, 2006 at 11:49 a.m. on the Rotten Eggs website in an article titled (amazingly enough): Hacking electronic road signs.  Of course the newest article is a little more in-depth, but this type of activity and vulnerability should not have been a surprise to anyone.  Those of you who subscribe here are very familiar with this type of situation.

Now that the situation is back in the public eye, how do we leverage it with our friends, family, co-workers, customers, and management?

What this situation does is emphasize the fact that default passwords and devices with built in reset capabilities should be controlled in a much better manner.  The changing of a road sign will not last very long or adversely affect (generally) anything beyond inconvenience.  The real problem is the mentality of companies creating devices that operate in this manner.  Things are still getting built this way and we have to make the logical leap that developers of hardware, programs, operating systems, network devices, mobile devices, and applications are making the same mistakes even today.

We can use this opportunity to remind our our friends, family, co-workers, customers, and managers to evaluate their deployed technologies for default passwords.  We should also remind them that they need to take these things into consideration during the initial purchasing process where they are evaluating new technologies. That is the only way to find these types of problems and mitigate the risk properly before purchasing and deployment.  Should they find devices or applications with these limited or hamstrung security capabilities they should do a risk assessment to determine the best method to increase the security surrounding the technology or what can effectively and securely replace it.

Your mission is to determine a way to put the preceding paragraph into words and terms that your audience will understand.  Most of you reading this know exactly what I talked about.  You cannot assume that your audience will be able to understand it in the same manner.  If they don’t understand it they cannot proceed effectively.  Think about your audience before approaching them with your recommendations.  Determine the proper terminology, references, and examples to help them make an informed assessment and conclusion.  Be prepared with solutions for situations that you know exist and methods to move forward and locating those situations that have not identified.  And be sure to stress the importance of taking security into consideration during the initial evaluation and purchasing process.

Go forth and do good things,

Don C. Weber

Can you identify the root cause?  What I mean is, if you were presented with a question, about how an event occurred, could you show the logical progression of how that event occurred?  And then describe to me the actions that followed?  Maybe you are not the network/system/security administrator of your organization.  Fine.  Can your people answer those questions for you?

The SANS Glossary of Security Terms defines an event as “an observable occurrence in a system or network.”  It also defines an incident as “an adverse network event in an information system or network or the threat of the occurrence of such an event.”  But I am not talking about an incident.  I don’t care if you have had an incident.  What I want to know is if your organization has the ability to determine the steps that lead up to and happened after an “event.”

Because if you cannot describe to me the circumstances surrounding an event, you cannot describe to me the circumstances surrounding an incident.  If you cannot tell me who initiated a connection to your web server from an internal asset, then you cannot tell me the actions that person performed after creating that connection.

Do you want to know what happened to your information or system at a certain time period?  Are you obligated to know by policy, regulation, or just plain common sense?   I can go so many different directions with this thought process.  Really, it all depends on the persons that I am talking to and their roles within the organization.

One of the first things that I tell people who are just starting out in the security industry is that security is really the art of implementing good business practices.  Although there are several specialized portions of the security industry, the majority of us are employed to ensure an organization is implementing sound and well documented methods to maintain a well-structured and profitable business.  Unfortunately, the thing about good business practices is that it is driven by continued profit.  And as we can see from the state of the economy, those profits are the very thing that can blind us to the affects of risk.  Or at least provide us with the motivation to accept the risk.  Of course I would love to believe my predictions from my last post about Conficker/Downadup – Securing The Internet as it is in my nature to trust these types of eventualities.  But most likely a writer over at VistaSpyware.com is correct in the assumption that it is not in the nature of the industry to do so.

Although you might think that I am straying from the “Root Cause” theme here I am not.  Because being able to determine root cause and follow on actions invariably depends on the methodology that you and your organization have implemented within your environment.  If you have been lazy or sloppy about patch and asset management then you are not going to be able to attest to the state of your infrastructure.  If you have decided to roll back logging and alerting because of the impact to storage space or network bandwidth instead of augmenting the technologies to handle the loads properly then you are not going to be able to attest to the state of your infrastructure.  Scaling back on the infrastructure that does not necessarily provide an immediately measurable positive monetary impact on your environment will eventually lead to a state that you cannot provide accurate and detailed information about the root cause and follow on actions of an event.

Certainly, a good incident response team can piece many things together.  Through hard work, knowledge, perseverance, and a lot (and I mean A LOT) of your money they will be able to provide you with the best report they can generate.  And sometimes the information will be helpful possibly finding the root cause and follow on actions.  But there are going to be other times when they cannot.  Sometimes they will finish the initial contact phone call by providing you with quick recommendations on how to proceed, thanking you for your time, and then gently let you know that there just is not enough information to proceed with an analysis.

Now, here is the part where I ask you if you want this to be you.  Well, actually, I am not going to.  You have already decided.  Or, it has already been decided for you.  What I mean by that is that the resources you have already been given are in place and you will more than likely not get anything else.  Or, at least, you need to begin thinking that way.  I say that because everybody needs to start thinking about the resources that they already have in place and start implementing them properly.  Start utilizing the technologies that have been deployed, determine how they work together and can augment each other, and then configure and manage them correctly.  Pull back and start asking yourself, your team members, and your organization if you are doing the basics correctly.  After you have done that, take a look at your more advanced technologies and determine if they are employed correctly to assist your organization in implementing the basics more effectively and efficiently.  Because that is how the more advanced technologies are intended to be implemented.  Until you have done all of this you should not be trying to find the next best security or business technology to act as your silver bullet or to help you “pull it all together.”  Because if you do not have it all together in the first place, dropping something more complex into the mix is going to move you further away from your ultimate goal of cost effective implementation, stream-lined business methodology, and low-risk high-yield technology to revenue ratios (not sure where that last one came from, but it sounds good so I’ll leave it in).

To summarize, if you want to know when, how, and why you or your organization got hacked you have to prepare before hand.  You have to employ basic business, technology, and security strategies thoroughly and correctly.  Only then will you get a good answer to the question, “What was the root cause of this event/incident and what has occurred since the initial occurrence?”

Go forth and do good things,

Don C. Weber

I have to say one thing for the rash of Conficker/Downadup infected systems that are plaguing businesses around the world:  This malware is helping the overall security of the Internet.

Once we are past this round of malware it will definitely be harder to propagate a mass infection.  Scoff if you will, but I am serious.  I admit that this worm is building a very large network of infected systems.  But for those businesses that are addressing this malware attack they are discovering the weaknesses within their infrastructures and response techniques.  The down-times associated with locked accounts, offline servers, disrupted services due to network traffic saturation, poorly implemented / broken patch management capabilities, broken backup procedures (BTW, are your backups infected?), etc are helping the information technology staff justify their recommendations to fix these issues.  Whether the recommendations were already in place or are now forth coming is really irrelevant.  The fact is that once businesses start adding up the costs associated with the response to this malware, executives should start taking notice of the potential return on investment (gasp, Securit ROI – opps, please don’t start THAT conversation AGAIN) of their network security and management technologies.

I am certainly not saying that after this malware tumbles off into the distance that there will not be another instance of a mass infection.  What I am saying is that because of the Conficker/Downadup malware, many organizations are going to be better prepared to avoid, limit, eradicate, and return to business as usual.  This will, in-turn, reduce the number of infected systems and the speed that new malware propagates across the Internet.  I wish I could also say that this will help other non-business computers such as those owned by schools, non-profit organizations, home-users, and [add your own choice here], but that is, unfortunately, just not the case.

One thing I can recommend is that IT staff and management take advantage of this situation and make their recommendations quickly with an emphasis on prioritization.  Recent disasters have shown how short lived memories associated with purchasing and implementing protections associated with business continuity and disaster recovery can be.  Determining which technologies will give you the most bang for your buck while also increasing your infrastructure’s preparedness with an emphasis on reducing the gap between an incident and the organization’s initial response is key.  Organize the rest of your list with these issues in mind.  Hopefully, you will get the number one priority on your list.  But if your list is not prioritized you may be stuck with a box of stuff that will leave you scratching your head and wondering how it is going to help future incident responses and general business requirements.

Go forth and do good things,

Don C. Weber

** It was recommended that I add a disclaimer stating I am not a lawyer.  So, be advised, I am not a lawyer **

*** Update 2: shrdlu points out (see comments below) that I missed the very last line of the PSB opinion on security consulting.  Thank you, shrdlu.  So, until they change that opinion all is well. ***

The SANS Computer Forensics Blog post Digital Forensics Professionals: Texas PI Legislation Interpreted got me thinking about the Texas PI laws again so I decided to take another look at the TEXAS OCCUPATIONS CODE CHAPTER 1702.  What I have found concerns me very much and if you or your company does forensic or security consulting work in the State of Texas then you had better read it as well and pass it onto your lawyers BEFORE you do anymore work in this state.

Basically, the State of Texas is now regulating all of the information security consultant industry within its boundaries.  This *DOES NOT* include the security departments of individual businesses.

Let’s start with a bit of clarification.  The State of Texas designates anybody doing investigative or security consultant work as a “company.”  If you do not understand this then reviewing the statue is going to confuse you at first.  With this in mind, here is the definition of “investigations” as put forth in Chapter 1702.

Sec. 1702.104.  INVESTIGATIONS COMPANY.

(a)  A person acts as an investigations company for the purposes of this chapter if the person:

(1)  engages in the business of obtaining or furnishing, or accepts employment to obtain or furnish, information related to:

(A)  crime or wrongs done or threatened against a state or the United States;

(B)  the identity, habits, business, occupation, knowledge, efficiency, loyalty, movement, location, affiliations, associations, transactions, acts, reputation, or character of a person;

(C)  the location, disposition, or recovery of lost or stolen property; or

(D)  the cause or responsibility for a fire, libel, loss, accident, damage, or injury to a person or to property;

(2)  engages in the business of securing, or accepts employment to secure, evidence for use before a court, board, officer, or investigating committee;

(3)  engages in the business of securing, or accepts employment to secure, the electronic tracking of the location of an individual or motor vehicle other than for criminal justice purposes by or on behalf of a governmental entity; or

(4)  engages in the business of protecting, or accepts employment to protect, an individual from bodily harm through the use of a personal protection officer.

(b)  For purposes of Subsection (a)(1), obtaining or furnishing information includes information obtained or furnished through the review and analysis of, and the investigation into the content of, computer-based data not available to the public.

After contacting the Private Security Bureau, which is a division of the Texas Department of Public Safety, I was told that the State of Texas regulates “investigations” so that the persons conducting them are qualified.  To ensure that investigators are qualified they are required to comply with Sec. 1702.113.  GENERAL QUALIFICATIONS FOR LICENSE, CERTIFICATE OF REGISTRATION, OR SECURITY OFFICER COMMISSION (which are basic employment requirements) and:

Sec. 1702.114.  ADDITIONAL QUALIFICATIONS FOR INVESTIGATIONS COMPANY
LICENSE.

(a)  An applicant for a license to engage in the business of an
investigations company or the applicant’s manager must have, before the
date of the application, three consecutive years’ experience in the
investigative field as an employee, manager, or owner of an investigations
company or satisfy other requirements set by the commission.

Now I do understand the need to provide a governing hand to protect the public from “investigators.”  If the state feels it is necessary then so be it.  This is basically stating that you have to have three years experience before you can operate individually or be the primary investigator of a company (yours or somebody elses).  More explaintion about this can be found in the PSB Opinion Summaries in the section titled Computer Forensics.

The part of Chapter 1702 that is really going to concern people is the guidance it provides for people the state considers as “security services contractors” or “private security consulting company.”  Here is the guide lines for what constitutes as a “private security consulting company.”

Sec. 1702.1045.  PRIVATE SECURITY CONSULTING COMPANY.

A person acts as a private security consulting company for purposes of this chapter if the person:

(1)  consults, advises, trains, or specifies or recommends products, services, methods, or procedures in the security or loss prevention industry;

(2)  provides a service described by Subdivision (1) on an independent basis and without being affiliated with a particular service or product;  and

(3)  meets the experience requirements established by the board.

Guidance on how this applies can also be found in the PSB Opinion Summaries in the section titled Computer Network Vulnerability Testing Firms.  Here is the part that stands out to me:

However, while the Bureau regulates consultants in the “security industry
or loss prevention industry,” these latter phrase is not explicitly defined
in the statute. It is therefore necessary to look to the rest of the
statute in order to understand to which services the private security
consultant’s licensure requirement applies.

It is reasonable to consider those industries otherwise regulated by the
Private Security Act as reflecting the scope of the phrase “security
industry or loss prevention industry.” In other words, the definitions are
implied by those services that are regulated by the statute, viz., security
guards, locksmiths, alarm system installers and monitors, and private
investigators, and not software designers, installers or suppliers.

Thus, the industries that are directly regulated are the same industries
about which one cannot consult without a license. Because the Private
Security Bureau does not regulate software designers, installers, or
suppliers, it also does not regulate those who provide consulting services
related to computer network security.

What this tells me is basically, if you are a security consultant in the State of Texas you must be registered.  This requires that you apply for a license and pass the Qualified Manager’s Exam.  This is the same exam that is required to become a licensed Private Investigator only where as Private Investigators only pay $350 to take the exam, security consultants have to pay $400 to take the exam, as outlined in Chapter 1702.  This exam simply shows that the person passing the exam has an understanding of the regulations we are covering and nothing specific to investigations or consulting.  The additional requirements to become a licensed security services contractor include:

Sec. 1702.115.  ADDITIONAL QUALIFICATIONS FOR SECURITY SERVICES CONTRACTOR LICENSE.

(a)  An applicant for a license to engage in the business of a security services contractor or the applicant’s manager must have, before the date of the application, two consecutive years’ experience in each security services field for which the person applies as an employee, manager, or owner of a security services contractor or satisfy other requirements set by the commission.

(b)  The applicant’s experience must have been obtained legally and must be:

(1)  reviewed by the commission or the director; and

(2)  determined to be adequate to qualify the applicant to engage in the business of a security services contractor.

As a security profession in the State of Texas I am concerned that I cannot consult, advise, train, or specify or recommend products, services, methods, or procedures in the security or loss prevention industry without being a licensed security services contractor.  This basically tells me that I cannot talk to anybody (family, friends, public gatherings like the PTA or a church, in addition to business relationships) about these issues until I am licensed.  Consultant businesses doing business within Texas should have the very same concerns.

Security professionals coming to Texas should also be concerned.  If you come to Texas to work or even to teach a class (SANS training) or give a presentation (TRISC) that consults, advises, trains, or specifies or recommends products, services, methods, or procedures in the security or loss prevention industry and you are not licensed you could be held accountable.  Specifically:

Sec. 1702.386.  UNAUTHORIZED EMPLOYMENT; OFFENSE.

(a)  A person commits an offense if the person contracts with or employs a person who is required to hold a license, registration, certificate, or commission under this chapter knowing that the person does not hold the required license, registration, certificate, or commission or who otherwise, at the time of contract or employment, is in violation of this chapter.

(b)  An offense under Subsection (a) is a Class A misdemeanor.

Although a Class A misdemeanor does not seem like much, individuals who have been found in violation of this statue may not be able to obtain a license in the future as outlined in Sec. 1702.113.  GENERAL QUALIFICATIONS FOR LICENSE, CERTIFICATE OF REGISTRATION, OR SECURITY OFFICER COMMISSION.

If you have additional information, updates, or clarification on this please leave a comment or shoot me an email so that I can update this post.

I’m starting to wonder if this blog is a violation of this statue.

Go forth and do good things,

Don C. Weber

Not all incident responses go according to plan.  This means that a responder has to have multiple methods for accomplishing tasks.  Harlan Carvey brings this up on his blog often and it is why Ed Skoudis created his famous WMIC Kung Fu write-up and turned it into a SANS Course: Windows Command-Line Kung Fu In-Depth for Info Sec Pros.

On a recent engagement this fact came into play.  I went on site knowing that I would have multiple types of Unix and Linux operating systems to potentially acquire.  From the customer’s description I also knew that I would be dealing with some older equipment but nothing from their descriptions seemed out of the ordinary.  Once I  got on site I reviewed the situation with the customer and everything seem normal.  As I was not worried about collecting volatile data from the system I requested the customer set up a maintenance window and temporarily take the systems off line in a manner that provided the least amount of impact to their operations.

All went well until the system administrator handed me the hard drive.  It all seemed fine at first until I tried to plug the drive into the adapter that was connected to my write-blocker.  The pins didn’t match up.  Although there were plenty of pins across, the layers were placed too close together.  It was fine for the blade server that it came out of, but none of my adapters could plug into this configuration.  At first I figured this was no big deal.  I decided to boot to a Linux boot disk and just copy the drive from there to a USB hard drive.  Well, there was two problems with that: 1. The blade server did not have a CD-ROM drive, and 2. the USB port was version 1.1 and would not recognize the external hard drive that I had purchased (no big deal because 12MB per second would not cut it either!!).  So, plan one == bust and plan two == bust.

So what was plan three?  Well, dump over NFS using native commands, of course.  Luckily the system administrator was well versed in NFS and the commands we were about to use.  So there was not a lot of explaining, I didn’t have to type commands, and my only real job (as it should be in this type of situation) was mindful and guided keystroke monitoring.  Here is what we did.

• To start off, please review this great guide on setting up NFS via the Ubuntu community wiki.  It does not dive deeply into NFS.  It just gives the basics on how to get it up and running on a live system or Live CD.  Of course, mileage may vary depending on distro, so beware.  Once it was all set up we navigated to the correct directories and prepared to acquire.  When all was done the system we were acquiring had a remote share mounted to /tmp/target.
• Before we started copying files we needed to keep track of everything we did.  Of course we could have done that as soon as we logged into the system, but as it was the system administrator performing the actions and this was not necessarily a “Live Acquisition” it was not completely necessary and we needed to have a good place, like an NFS mounted directory, to copy our files.  So we changed to the /tmp/target directory and ran the script command to record our actions and sent it to an appropriately titled file.
• Next we needed to record the system time.  Although we did note the BIOS time when the system was booting up, we need to record the time displayed by the operating system.  Running a quick date command proved enough.  No redirection was necessary as we were already recording everything into a file.
• We needed to know what type of system we were running our commands on.  A quick uname accomplished this.  Of course different distros also have different methods of providing version information about themselves.  Look to website for the distro you are working with to determine which file to check in and then cat that information so that it is displayed to the screen and thereby written to your script file.
• In order to get all of the system information written to the correct directory we needed several commands.  Unfortunately, if the system had been compromised and a rootkit had been deployed, there was the potential that critical information would not be copied.  To help alleviate some concern behind this issue we decided to run a few commands.  First we needed to know where each of our commands were located.  This was determined through a simple which command.  This let us know where the system thought the program was located and allowed us to call it directly as we did not want to rely on the $PATH environment variable.  Next we needed to get the versions for each command so that we knew which versions of the commands we were working with in case of any problems or other issues such as non-standard programs.  Most commands will take either a –version or -V and provide specific details about the command.  In this case we needed to know the version of md5sum, dd, and split that was provided by the system.  Once this information was displayed we had it recorded.  Next we took the md5sum of each of these commands just to be sure.  If necessary these outputs could be checked against known good versions from known good sources of the distro.  If really paranoid and the tools are available, other hashing tools could be used.  Although all of these tools could be “rooted” it is not very likely.  Plus, in this instance, we were not too worried about a rootkit, but you never know.
• Now were were ready to copy the information to our mounted directory.  The hard drive was recognized as hda by the operating system.  This means that we needed to copy /dev/hda using the dd command.  But since we were copying a large hard drive (80 GB +) we needed to use the split command to help with errors.  Now, I did not know this before we started, but I heard from one of my co-workers and it was confirmed by the system administrator, that copying large files (everything is a file in *nix) to a NFS mounted directory was not completely reliable.  To help control this I was told that copying to 2GB files would be the best way to handle it.  If something failed then it would be easier to pick up from the last good file and just calculate the proper offsets to start the copying process from  that point instead of the beginning.  Here is the command we ended up running: /bin/dd if=/dev/hda | /usr/bin/split -d -b 2000m – /tmp/target/server_name.dd. Although you can review the split man page to understand the full command, I will point out two things.  The dash “-” is necessary because it tells the command to use the STDIN from the pipe as the input file.  The “.” at the end of the command is also necessary because it is what split uses to number the files it creates.
• Depending on the system this will run for quite a while.  But before you run off for coffee and donuts or to analyze another system you’ll want to run one more command.  That command is the date command.  To do this just type date and press enter.  This will automatically run the date command once the previous command has completed running.  This time and date gets recorded into the script and the full run time is recorded for use in reporting and future efforts.

Although all of this took some effort to do, the end result was that I had acquired the system in a manner that I could use multiple tools to evaluate the information.  Once this was all complete I moved onto the second webserver where we had similar issues with acquiring the hard drive as the system was running on the same blade server.  I would love to tell you that I just followed my previous steps and that all went well.  Unfortunately I cannot.  We did perform all of the steps but as soon as we hit enter on the dd command the system virtually locked up.  This came as quite a surprise as the previous system did not take a significant performance hit during the acquisition.  We stopped the command and did quick review of the logs on this new system.  The log was full of I/O errors and we could only assume that the hard drive was slowly failing.  Therefore, plan three == fail.  That is where the system administrator came to my rescue.  As the system did not have any problem backing up every night we decided to just tar the operating system over to the NFS mounted directory.  Although this would not get everything off of the drive, it would at least get some of the information and some is better than none, at least at this point. Here how the tar was accomplished.

• Follow all of the previous steps to get the information about the system and the commands.
• Next you need to understand the version of tar provided by the distro you are working with.  Read the man page just to be sure you have the options determined correctly.  Because of the way that tar interacts with NFS (thank you, schism) it is probably going to be necessary to use the “–one-file-system” option.  This means that you will have to acquire each of your mounted directories separately.  For instance, if you have one partition mounted at /, one at /boot, and one at /home the “–one-file-system” command will not follow down the mounted directories.  You will have to do each separately.  The following command worked like a charm and then repeated for each mount point.  /bin/tar –one-file-system -cvf /tmp/target/server_name_root.tar /

So, after four tries I had all of the information that I needed to start my analysis.  Well, at least for these systems.  This simple acquisition really threw me for a loop initially.  Not only did I have to pull out multiple acquisition techniques it also affected my time on site.  That might not seem like much, but these extra step meant that I was on site for an additionally two days.  No small impact on the customer.  But, it could have been a lot worse.  If I had not been prepared with fall back alternatives, then I could have spent even more time on site.  Not only would this have impacted the customer’s pocket book but it would have also sewn the seeds of doubt into my capabilities.  This could have, in turn, jeopardized their view of the results of the analysis as well as the potential for the continuation of a forensic, or any other, business relationship.

Go forth and do good things,

Don C. Weber

Jeremiah Grossman has a simple but sweet post about what to do on your first day of work when you come on board to a company that has no “no existing web/software security program.” He simply asked, “What is the very first thing do on day 1? [sic]”

The meat of the post is in the comments. Although it started out with some typical guidance on how to technically identify server, applications, vulnerabilities, and the like, the comments quickly transition into focus on the people of the organization. Getting to know the executives, management peers, security and technical administrators, and even support personnel before diving in and trying to find problems and giving orders about how to fix them.

Security Professionals need to remember that there are other people out there. It has often been said that we need to refrain from saying “No,” “Don’t,” “Can’t,” and other negatively connotative words unless absolutely necessary. We often remind ourselves that we are a part of the business unit and that we are, typically, support personnel rather than the front line administrators (and if you are both then your security tasks should take the support model into consideration). So when it all boils down, we are saying that we have to be a helpful and viable part of the business by working with the other employees, no matter the level, rather than being the lonesome cowboy with six-guns drawn. Once we have accomplished this then we can start delving into identify critical physical assets, location of data, mission critical application, and other important technically-related security information. Hopefully, your initial dealings with fellow employees and managers will have already greased the skids to start working with this information, but it will have also provided you with a better understanding of the politics and business necessities surrounding the current state of technical deployment.

I’m not going to repeat my or anybody else’s comments here. Go check out Jeremiah’s post and then put in your two cents. But while you are there, notice some of the names of people who are commenting on getting to know the people and organization first before diving into the technical aspect of the position. You will probably notice many people that you know and respect.

Go forth and do good things,

Don C. Weber

I would like to start off by saying, “You can’t!!” The quicker you come to grips with that the better off you will be in the long run. Politics, or perhaps Micro-Politics since I am talking about intra/inter-office politics, is just a fact of life. Everybody has an agenda whether it is to further themselves, further their family, further the company, or any number of other things. So, get over it because it is just going to happen.

Now, let me tell you how you can control politics. I’m not talking “hand of God” control. I’m talking about making it difficult for politics to adversely (because some politics are good) influence the security of your organization. The answer can be found in my previous post on Organized Security. The answer is “Document Your Processes!” Okay, that is not the full answer, but it is the start. Getting your processes written down and accepted is the first step. The thing that seems to be working the best for my team is to document a process’ flow before writing down the procedure. Understanding the actions, decisions, and touch points of a process before writing the document that details each action and decision point. Here is a simple example pertaining to a user account request. This process flow utilizes “swim lanes” to show different teams or departments.

Once you have created this flowchart it is very hard to justify a deviation from this process. It becomes even more difficult once you detail each box in your procedural documentation. Getting your management and each team or department listed in the “swim lanes” to sign off on their involvement with the process will decrease the deviation possibilities even more. And if all else fails, it will make deviations readily apparent to management and all of the teams or departments involved.

Now, this does not mean that deviations will not happen. It is a fact of life that a situation or event was not taken into consideration during the development of the process. These instances shouldn’t matter in the grand scheme. Once the event has happened and been addressed, the individuals responsible for the process should quickly run through the process to see if any documentation needs to be generated or additional actions taken. After everything has been addressed the team can conduct a lessons learn to determine if the process needs to be updated or if the deviation was just an anomaly that will rarely occur and can be addressed on a case by case basis. Of course, politics can fall into this category. But all of this, as I mentioned, makes the deviation very apparent and the extra work associated with running back through the process and evaluating the overall process should raise questions about the validity of the action.

Once everything is documented and approved there is another very important step. That step is to consistently apply the process. Lack of consistency will leave gaps in all of your processes. Lack of consistency will breed contempt for your system and provide individuals and groups the leverage they need to circumvent the process in question and possibility the other processes developed by your team.

In the end you are not going to solve politics in your organization. You and your team need to learn how to accept it as a part of doing business. Just remember, diligent documentation, repeatable processes, and consistent application will protect you as much as they can.

Go forth and do good things,

Don C. Weber

Well, this week at work was very interesting. Actually, the last two weeks have been extremely busy. As Friday rolled in I looked into the eyes of my team members and I could see the tired, slightly overwhelmed, and, for some, haggled look in their eyes. They had shouldered what our organization decided to throw at them and they pulled through with their heads held high. No small feat when you are talking about a crew with that was built from individuals with very little security background and a manager (me) who is hell-bent on documenting and improving each procedure as they are going through it. I do this not only to help them build a program that is repeatable and lends itself to self-improvement, but so that our customer can “feel-the-pain” when their goals are not being accomplished due to the never ending “high priority” additional tasks (something I, and others, refer to as “firefighting”).

I usually make it a point to congratulate my team members for a job well done. It builds confidence, denotes achievement, and helps give a sense of closure to on-going tasks and issues that never seem to have an “end.” But this week I went a step further. I let them know that when they are working on the “high priority” issues, when the “firefighting” is taking all of their time and effort, that the things they are doing are enough. Just working the task is enough to help secure our environment. Even if they haven’t completed the task or specific issues mean they were not able to address regular duties and other tasks, as long as they worked hard and smart, it is enough.

It has to be enough. No environment is ever going to be 100 percent secure. Security professionals and security cynics can all agree to that statement. But, when you look at it from the other end, no environment is zero percent secure either. Each operating system comes with some controls. So every environment starts a little bit “in the black.” As an organization starts adding personnel and controls they increase their security percentage. Finally, with the addition of security professionals and a well-rounded security approach, an organization sees its greatest jump towards the unobtainable 100 percent secure goal. Just dong things to move towards that endpoint is enough. And I think that sometimes organizations and managers forget that aspect of the big picture.

So, when you get back in the office next week, take a look around. Look at the accomplishments of your team members. Take note of these accomplishments and provide the appropriate praise to the situation. Let them know that their efforts are enough and that because of their actions the overall environment is more secure. Then look at the other individuals in your organization. Look at the system administrators, the desktop support personnel, the help desk operators, and everybody else. Look at their actions and point out their accomplishments as well. Let them know that they are helping secure the environment and that their actions are enough.

If you do this, you are doing enough and you are speeding up your progress towards that unobtainable goal.

Go forth and do good things,

Don C. Weber