Security Ripcord

While reviewing some Apache log files the other day I started to wonder if somebody had already come up with a way to detect common attack characteristics by the information they contain.  Although searching for entries containing “SELECT,” “xp_cmdshell,” and other attack terms can turn up useful information it is difficult to cover all of the attack types and the different ways they can be represented within the log entries.  Some Googling brought me to a project started by Romain Gaucher called apache-scalp which is hosted on Google Code.

Scalp! is a log analyzer for the Apache web server that aims to look for security problems. The main idea is to look through huge log files and extract the possible attacks that have been sent through HTTP/GET (By default, Apache does not log the HTTP/POST variable).

This tool uses an event filter file created by and for the PHPIDS project.  PHPIDS was created parse web traffic and alert on anomalous activity as it occurs instead of having to parse the log files.  To help in those instances that PHPIDS is not deployed, Scalp applies the same principle to the collected information.  Utilizing the PHPIDS signature file, default_filter.xml, Scalp will generate an alert file in the user’s choice of text, HTML, and XML formats.  This output, however, is alert data only and can be very cumbersome to review.  When a lot of information is involved human review is not a very effective way to determine trends and other specifics about the data.

<?xml version=”1.0″ encoding=”utf-8″?>
<!–
File created by Scalp! by Romain Gaucher – http://code.google.com/p/apache-scalp
Apache log attack analysis tool based on PHP-IDS filters
–>
<scalp file=”apache_log” time=”Sat-27-Dec-2008″>
<attack type=”xss” name=”Cross-Site Scripting”>
<impact value=”5″>
<item>
<reason><![CDATA[Detects JavaScript with(), ternary operators and XML predicate attacks]]></reason>
<regexp><![CDATA[(?:with\([^)]*\)\))|(?:\.\s*source\W)|(?:\?[^:]+:[^;]+;)]]></regexp>
<line><![CDATA[xxx.28.xxx.249 - - [26/Aug/2008:00:00:13 -0700] “GET /d.AuthenticateUser1?p_page=http://webx.companyX.com/publish/01/na/en.html&p_HTTP_USER_AGENT=Microsoft%20Internet%20Explorer-4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20SV1;%20.NET%20CLR%201.1.4322) HTTP/1.0″ 200 1547
]]></line>
</item>
<item>
<reason><![CDATA[Detects JavaScript with(), ternary operators and XML predicate attacks]]></reason>
<regexp><![CDATA[(?:with\([^)]*\)\))|(?:\.\s*source\W)|(?:\?[^:]+:[^;]+;)]]></regexp>
<line><![CDATA[xxx.16.xxx.158 - - [26/Aug/2008:00:00:19 -0700] “GET /d.AuthenticateUser1?p_page=http://webx.companyX.com/publish/01/na/en.html&p_HTTP_USER_AGENT=Microsoft%20Internet%20Explorer-4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20SV1;%20.NET%20CLR%201.1.4322) HTTP/1.1″ 200 1570
]]></line>
</item>
<item>
<reason><![CDATA[Detects JavaScript with(), ternary operators and XML predicate attacks]]></reason>
<regexp><![CDATA[(?:with\([^)]*\)\))|(?:\.\s*source\W)|(?:\?[^:]+:[^;]+;)]]></regexp>
<line><![CDATA[xxx.84.xxx.90 - - [26/Aug/2008:00:00:41 -0700] “GET /d.AuthenticateUser1?p_page=http://webx.companyX.com/publish/01/vie/en.html&p_HTTP_USER_AGENT=Microsoft%20Internet%20Explorer-4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20SV1;%20.NET%20CLR%201.1.4322) HTTP/1.1″ 200 1568
]]></line>
</item>
<item>
<reason><![CDATA[Detects JavaScript with(), ternary operators and XML predicate attacks]]></reason>
<regexp><![CDATA[(?:with\([^)]*\)\))|(?:\.\s*source\W)|(?:\?[^:]+:[^;]+;)]]></regexp>
<line><![CDATA[xxx.160.xxx.243 - - [26/Aug/2008:00:00:57 -0700] “GET /d.AuthenticateUser1?p_page=http://webx.companyX.com/publish/01/par/en.html&p_HTTP_USER_AGENT=Microsoft%20Internet%20Explorer-4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20SV1;%20.NET%20CLR%201.1.4322) HTTP/1.1″ 200 1573

To help identify trends and other interesting information associated with alerts generated by Scalp I have put together a external parser to generate a readable report.  The Scalp External XML Reporter (SEXR) takes the information within a Scalp XML file and produces several outputs to either standard out or a text file.

The first output is the full parse of the generated alerts.  Although this output can still contain a lot of information, it is much easier to read and understand than the original Scalp output.  (The following output, as well as the rest in this post, have been snipped for brevity and their format slightly altered getting the text into this post.)

sexr.py: Conducting full scan of 1 files
scalp: {‘file’: ‘apache_log’, ‘time’: ‘Sat-27-Dec-2008′}
attack: {‘type’: ‘xss’, ‘name’: ‘Cross-Site Scripting’}
impact: {‘value’: ‘5′}
item
reason
- Detects JavaScript with(), ternary operators and XML predicate attacks
regexp
- (?:with\([^)]*\)\))|(?:\.\s*source\W)|(?:\?[^:]+:[^;]+;)
line
- xxx.28.xxx.249 – - [26/Aug/2008:00:00:13 -0700] “GET /d.AuthenticateUser1?p_page=http://webx.companyX.com                    /publish/01/na/en.html&p_HTTP_USER_AGENT=Microsoft%20Internet%20Explorer-                        4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20SV1;%20.NET%20CLR%201.1.4322) HTTP/1.0″ 200 1547

item
reason
- Detects JavaScript with(), ternary operators and XML predicate attacks
regexp
- (?:with\([^)]*\)\))|(?:\.\s*source\W)|(?:\?[^:]+:[^;]+;)
line
- xxx.16.xxx.158 – - [26/Aug/2008:00:00:19 -0700] “GET /d.AuthenticateUser1?p_page=http://webx.companyX.com/publish/01/na/en.html&p_HTTP_USER_AGENT=Microsoft%20Internet%20Explorer-4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20SV1;%20.NET%20CLR%201.1.4322) HTTP/1.1″ 200 1570

item
reason
- Detects JavaScript with(), ternary operators and XML predicate attacks
regexp
- (?:with\([^)]*\)\))|(?:\.\s*source\W)|(?:\?[^:]+:[^;]+;)
line
- xxx.84.xxx.90 – - [26/Aug/2008:00:00:41 -0700] “GET /d.AuthenticateUser1?p_page=http://webx.companyX.com/publish/01/vie/en.html&p_HTTP_USER_AGENT=Microsoft%20Internet%20Explorer-4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20SV1;%20.NET%20CLR%201.1.4322) HTTP/1.1″ 200 1568

Although helpful and informative it might be a little more helpful to know more specific information about the alerts.  SEXR’s count scan option is designed to remove some of the more extraneous information by providing the number of alerts detected and the names of the alerts with which they are associated.

sexr.py: Conducting count scan of 1 files
scalp: {‘file’: ‘apache_log’, ‘time’: ‘Sat-27-Dec-2008′}
attack: {‘type’: ‘xss’, ‘name’: ‘Cross-Site Scripting’}
Impact 5 Items: 299
- ‘Detects JavaScript with(), ternary operators and XML predicate attacks’: 248
- ‘Detects self-executing JavaScript functions’: 51
Impact 4 Items: 655
- ‘Detects common XSS concatenation patterns 1/2′: 655
Impact 3 Items: 80
- ‘Detects common comment types’: 80
attack: {‘type’: ‘lfi’, ‘name’: ‘Local File Inclusion’}
Impact 5 Items: 199
- ‘Detects specific directory and path traversal’: 199
attack: {‘type’: ‘rfe’, ‘name’: ‘Remote File Execution’}
Impact 5 Items: 383
-’Detects url injections and RFE attempts’: 383
sexr.py: Done

After determining the types of attacks that were detected from the Apache log files it might be interesting to know where these attacks originated.  SEXR’s source IP scan option provides the source IP address for all of the alerts detected and a count of how many times these source IP addresses were associated with the attack.

sexr.py: Conducting IP scan of 1 files
scalp: {‘file’: ‘apache_log’, ‘time’: ‘Sat-27-Dec-2008′}
attack: {‘type’: ‘xss’, ‘name’: ‘Cross-Site Scripting’}
Impact 5 Items: 299
- Total Source IP Addresses: 209
- xxx.176.xxx.42: 1
- xxx.177.xxx.68: 1
- xxx.129.xxx.3: 1
- xxx.89.xxx.250: 1
- xxx.179.xxx.235: 1
- xxx.253.xxx.222: 3

Although Scalp does all of the heavy lifting I believe that SEXR can play an important role in evaluating the results that Scalp produces.  I hope that SEXR will help make the information provided by Scalp more usable for system, network, application administrators and security professionals.  To this end Romain has allowed me to update a few portions of Scalp to make its XML output more informative while also adding SEXR and the Scalp DTD file to the source code available online with the Scalp project.

If you have any comments, recommendations, or updates for the code, please let me know.

Go forth and do good things,

Don C. Weber

Reviewing my daily blog hits I came across another interesting search that directed somebody here.  “what a firewall and ids can tell you about an incident” Although I sarcastically commented on my Interesting Search Keywords page “That you need to turn on more logging” I have to admit that I meant it.  Firewalls and Intrusion Detection Systems will only provide the analyst (be it a professional incident responder or a system/network administrator) with a limited amount of information.  The real meat of an intrusion is going to come from the coalition of data from disparate logging resources.

• Firewall logs, depending on how logging is configured, is going to tell you about the connection attempts in and out of the network.  Most likely you are going to see the traffic that failed.  If the firewall has proxy capabilities you will be provided with specifics about the proxied traffic, also most likely the stuff that failed.  Depending on where the firewall is located within your infrastructure will determine how much noise will be present within the logs.  Internet facing firewalls will have a lot of noise and will therefore probably have failed logging toned down.  Internal firewalls, however, should not see much failed traffic.  So anything that shows up could protentially be interesting.  Comparing these logs could also be affective in determining the origin of specific attacks, were they intitiated internally or externally.
• IDS logs could be helpful or noisy as well.  These really depend on how well the security or network administrator has maintained the IDS sensors and where they have been placed.  Having every signature and capability turned on will lead to quite a bit of false positives to sort through.  But too much tweaking could leave administrators oblivious to reconnaissance efforts.  IDSes are great for determining if attackers are using known attack vectors to gain a beachhead within an environment.  But, like anti-virus engines, they are only as good as their signatures no matter how well the community and commercial companies stay on top of it.  I’m willing to bet it is going to be a long time before we see signatures for Clickjacking (BTW, Jeremiah Grossman recommends this article).
• These points aside, having the logs from both of these devices provides you with the best information of all.  What is the normal operational state of communications within the environment? There are no if, ands, or buts about it.  If you do not know what can be considered normal there is no way to asertain what is abnormal.  Having good logs from these devices will enable you to pin-point a specific time period that reconnaissance or even an attack occurred.  This could help you narrow your window of other investigation items be it system logs, file assess or write times, or registry modifications.

I said it on Twitter the other day, ” Logs are interesting, logs are fun, logs should be done by EVERYONE…..get to logging!!!” and I meant it.  It is very hard to investigate an intrusion or incident if there are no logs to review.  Bringing together the firewall logs, IDS logs, application logs, operating system logs, and anything else there is provides the foundation from which to build the response.  Without them you will just be staring at each other wondering what to do next.  And I’ll tell you what to do if you don’t have any logs.  Contain the incident, investigate as much as you can, return to a good operational state, disclose if you have to disclose, and TURN ON LOGGING.

Go forth and do good things,

Don C. Weber

I was going to post a comment to Michael’s Computer World post titled “OK CXO, does this incident convince you of the need for security???” when I decided not to do it. Instead I realized that this is a prefect opportunity to do another Email Interview. So I hammered out a few questions I figured would be easy to answer and then tossed them his way. The following are his responses with the questions included in bold.

Who do you work for and what do you do?

Accuvant. Pre-sales Security Engineer. Basically, I am in customer
relations, with the occasional product evaluation installation thrown
in.

Centralized logging is one of the keys to a good security posture.
Percentage-wise, how many companies do you see doing this?

Unfortunately, the customers I have been dealing with are very late
coming to this game. And I am not talking about SIEM either. Just
centralized logging. Many of them have some kind of syslog server with
a few logs getting thrown to it, but very few have any kind of real
centralized logging solution where they can go do forensics and get a
good idea of what was happening in their network as a whole at any given
time.

When they are designing the networks, do you find that the
administrators are aware of the pitfalls and nuances of setting up a
logging infrastructure?

Generally they think it is just a matter of throwing some logs into a
bucket o’ hard drives and that is it. Not many think about the logs
being used for forensic purposes later in the case of an incident. Many
aren’t aware that normalizing logs pretty much makes it fodder for the
defense attorney. Administrators up until now have had to be concerned
with keeping servers going, and they read the logs when there is a
problem. Most don’t think from a security mindset, so they don’t have a
clue what to do to if those logs aren’t there (because they got deleted
by Mr. Bad Guy). There’s also the matter of retention and drive space,
maintaining the logs in such a manner that they can’t be altered, etc.
I mean, that is the central reason for logs: forensics. And forensics
does not necessarily mean criminal forensics. It means that if there is
any incident, malicious, accidental, mechanical, whatever. And if the
logs get corrupted, then you have problems.

Another problem that people don’t think about is application logs and
“x”-flow data. These are often very critical to determining what
happened in incidents because they give you two ends of the spectrum
that just server and device logs don’t give you. Of course, that will
greatly increase your storage needs, so be careful.

Where are the common weaknesses and how can we educate our
administrators better? Certifications associated with log management
and review?

See above for common weaknesses. As far as another specialized cert, I
don’t think that is the way to go. I know SANS is big on this, and I
have nothing but respect for those guys, but I really think that
security cannot be stovepiped anymore. Security has to be a part of a
sys admin’s job and training. This is one facet of that training, and
it realistically not all of security can be thrown into a couple of
Windows courses. But the mindset has to be taught more. A single cert
can’t do that.

Is it common practice to have critical systems administered through a
management network that does not touch the production network?
Basically, a network that is separated from the intranet and Internet?

Do you mean having something like separate NICs in critical servers
plugged into a different VLAN for management, and plugging management
NICS on devices into that same VLAN? To my knowledge, that is not very
common. It makes sense, but I could see it being something of a
headache to get setup. As far a segmenting critical systems altogether,
that is happening in a big way. PCI is driving that everywhere.

Can Small and Medium-Size Businesses really afford taking on a project
like centralized log management? What would be the first steps?

Yes, SMB’s can do this. I does not have to be expensive. I did it when
I was at an SMB. I used the PRO version of KIWI syslog server
(http://www.kiwisyslog.com/) and pointed all my devices and servers at
it. I used SNARE (http://www.intersectalliance.com/projects/Snare/) to
push server logs to the KIWI syslog. KIWI even has the ability to read
application logs if they are put into a flat file. It really is not
hard to setup if you are willing to take the time to organize it. But
be sure to take note of the problems mentioned above.

Do you have any recommendations for the Small/Home Office businesses
when it comes to log management?

See above. Should also work for them.

Companies are generally aware that they need to backup their common and
critical data. How much are they aware that they need to do the same
with logs? What is common practice in log retention?

I think they are becoming MORE aware, but the awareness is not where it
should be. Again, this is a failing in training. Security is not
taught as a “baked-in” component of knowledge.

Common practice depends. It seems like 7 years tends to be a good
retention length of time, but that can change depending on compliance
and other laws.

What types of sensitive information could be found in logs and what does
this mean about the protections associated with collecting and retaining
this information? When do you think encryption would be necessary?

It is feasible for access logs on servers containing data to hold
sensitive information such as credit card numbers, SSN’s, etc. It is
also feasible for logs from network devices such as routers and
firewalls to have sensitive data in them because the data passes through
them. That is why PCI specifically addresses this issue.

Encryption of sensitive fields in a database should always be in place.
Encryption when transmitting data should be the norm. And obfuscation
of sensitive data (“x”ing out most of the SSN number or the CC number)
should be done when records are viewed by parties that do not need the
information to perform their duties.

How does log management tie into forensic investigations?

As I mentioned above, log management is crucial to forensic
investigations. Most log management systems normalize logs because they
have to store the data in a manageable format. However, there has to be
some way to recover the raw log in order for the log to be used in
investigations. It is not as crucial if all you are trying to do is
determine what happened in an event. But if plans are to use the logs
in prosecution of criminal activity, then you have to be able to prove
that the logs are the raw, original logs that came from the servers and
devices.

If an investigation team walks in the crime scene and fouls evidence,
the judge is very likely not to admit it. Or if the cops don’t retain a
good chain of custody of the evidence, then the defense lawyer can make
the claim that there is no way to know that the police did not alter the
evidence in some fashion. Logs are evidence, so the same rules apply.

Do any of the forensic programs out there integrate with the logging
solutions you are familiar with?

Good question. I don’t know of any that do off the top of my head.
However, if standards are used to code each one, there is likely a way
to integrate them and transfer logs between them. But again, this can
only be done successfully if the evidence is not altered in doing so.

How can your company help with the issues related to this topic?

Accuvant has four practice areas (Security Assessment, Compliance,
Security Technologies, and Wireless). Our compliance team would be
specifically helpful with this kind of issue because they know the
different compliance controls that would require a company to adhere to
log management standards, and they could perform a gap analysis to let a
company know where they are lacking. They can also provide remediation
assistance to fill the gaps.

The assessment team can perform policy review and architecture review to
tell a company where they are lacking in this area and their security
posture as a whole. They can also perform penetration and vulnerability
tests to see what is getting logged and at what level. And they can
perform application code review to determine if an app’s logging is
sufficient. They can also provide remediation assistance.

The security technologies team can help in recommending an appropriate
log management solution, and they can provide professional services for
installation, configuration, and management of the solution.

The wireless team does not seem to be specific to this issue, but
wireless is actually an area that many administrators and managers tend
to kind of forget is in their network. Thus, logging rarely happens
from these devices. Our wireless team has expertise with many different
manufacturers of wireless technology, and they would be able to provide
best practices for gathering logs from these devices.

You are a blogger and Internet author. Tell us a bit about that.

Internet author?? Really just a blogger since I have rarely written
anything more than a page at one time. But oh well. It sounds good!

I started blogging a little over a year ago when I was in the trenches
as an Information Security Manager at a small / medium-sized psychiatric
clinic in Houston. I really enjoyed being able to write about the
things I ran into during the course of my job. It gave me a chance to
grow as a security professional, and it really helped me hone my writing
style.

As I went forward with it, I realized that it was also a way to get my
name known out in the world of security, which I think will help my
career in the long run.

I maintain two security-related blogs. My personal blog is at
http://infosecplace.com/blog. That one is known as An Information
Security Place. I also blog about security at ComputerWorld at
http://computerworld.com/blogs/farnum.

I also have a personal blog at http://infosecplace.com/tangential. I
don’t update that on a lot. It is mainly a place to keep anything
personal I want to write about, and it helps me maintain a more pure
security blog.

Nothing really ground breaking here. Just a few questions that I thought might help others that are thinking about centralized logging. Hopefully Michael’s answers will help you when you are considering initiating or increasing logging and log management within your environment.

I would like to thank Michael again for taking time out of his very busy schedule to answer these questions. Check out his sites when you have a few extra minutes. It is definitely worth bookmarking or placing into your feeds.

Go forth and do good things,
Cutaway

logging, Accuvant, An Information Security Place, Computer World, Michael Farnum, Security Ripcord

Okay, so the title is a little bit of a misnomer.  I have not found any security websites using web bugs.  Where this stems from is my own pondering.  At my 8 to 5 organization I have been wondering how I should track the usage of the security based website I manage.  This makes good sense because I want to see if I am reaching my audience. 

As I am not the administrator of the resource I do not have the proper privileges to view the logs associated with the web server.  Actually, I do not want those privileges unless necessary for auditing or incident response purposes.  I also do not want to burden the system administrator any more than necessary.  So, rather than ask for the output of the log associated with my virtual host I have started thinking about methods that I can track hits using some type of PHP counter.

While I was investigating how to accomplish this I received a few emails so I took a look.  I had received a couple mundane messages and an email from a vendor.  Once I opened the vendor email I noticed the usual vendor email format which, unfortunately, is strewn with plaintext HTML links.  The very first link was a web bug.  It said so right on the image “title=”Web Bug…”.  So, I started thinking to myself,  “Hmmm, web bug.”  Would that work?  I could include it in every web page.  I could include it in all of the documents and presentations I provide on the site.  This will tell me how often stuff is getting viewed and whether they are coming from the intranet or the Internet.  Exactly the information I could get from the web log.  But for some reason the idea of including a web bug on the site and in the documents made my skin crawl.  So I decided to do a little asking around in the Security Catalyst Community.  I started a thread titled “Web Bugs on Internal Security Sites“.

My call was answered by several people including Rebecca Herold.  She provided me with some good insight but even better documentation.  First she pointed me to a paper by the National Advertising Initiative.

Also, the The National Advertising Initiative (NAI) created a set of standards that cover the use of web bugs (also called web beacons, web gifs, and several more a.k.a.’s) on Internet sites.  You may find their standards interesting and perhaps helpful to your consideration of using them within your network: http://www.networkadvertising.org/networks/Web_Beacons_rev_11-1-04.pdf

Next she posted a white paper that she wrote about web bugs back in April 2005 titled “Quit Buggin Me!”  I have read it already and I highly recommend it if you are interested in web bugs.  Although I usually link directly to a document I would rather force you to her site so you can be aware of her other papers and books.  The paper can be found at her “Articles Regarding Technology Aspects of Privacy” page.

By this time I was completely squared away by Ms. Herold.  Although not necessarily bad, web bugs are not necessarily good.  In fact, they have to be used properly or you may face issues with your users, your organization, or even your government.  In this case my government would be the State of Texas.  So, to continue my research I did a little Googling.  And, of course, I got a hit that directed me to the Texas Department of Information Resources.   It was very quickly apparent that the State of Texas has a policy on how to use persistent cookies and Web Bugs.  In fact, the guidance set by the Texas Department of Information Resources states:

In order for visitors to make informed decisions about the privacy practices of state agencies, the visitor should be able to access the home page and Privacy and Security Policy page without the site setting a cookie or using a web bug to track visitor [sic].

Delving in a little deeper I noticed that there is specific guidance for Institutions of Higher Education.  Particularly Texas Administrative Code Rule 206.73 Privacy and Security of State Web Sites.  

(a) Each institution of higher education shall publish a privacy and security policy for its Web site, and post a link to the policy from its home page, or Site Policies page. The privacy and security policy shall address the following:
  (1) Notice: This section must disclose the institution of higher education’s information practices before the site collects personal information from the public, including the use of, cookies, and/or Web bugs as well as information collected by other technologies and processes, and information collected via e-mail and Web-based forms.
  (2) Choice: This section must disclose whether and how personal information collected from the public may be used for purposes beyond those for which the information was provided.
  (3) Access: This section must address the procedure under which an individual may obtain information about himself or herself from the institution of higher education and/or have the institution of higher education correct information about the individual.
  (4) Security: This section must describe the procedures that ensure that information collected from individuals is accurate and secure from unauthorized use.

So, basically, after a little help from the Security Catalyst Community and a little research into the laws and regulations set forth by my government I have decided that it will be much better for me to glean the personal information of the visitors to my internal website from the web logs provided by the web server than to glean them from a web bug or some other type of overt tracking mechanism.

Go forth and do good things,
Cutaway

Powered by ScribeFire.

cookies, privacyguidance.com, DIR, SCC, Texas, Security Ripcord, Rebecca Herold, web bug