Security Ripcord

Cutaway Security Blog and Projects

Monitor Changes in OS X Using Fslogger-yaml

For the “TL;DR crowd,”: fslogger-yaml project page.

The release of EmPyre caused me to start thinking about the Indicators of Compromise (IOC) associated with it being used to interact with a system. The network artifacts, while they do need to be investigated, seem fairly straight forward to me. I am more interested in the system-based IOCs.

Since I’ve been out of the forensic arena for a while, I quickly realized that I have no idea how to track changes to system running OSX. There are, of course, system logs and Apple System Log (ASL) files. These do provide valuable information but they do not track everything. I wanted something that would provide me with enough detail that would assist in the generation of consistent IOCs.

I spent the weekend doing some research on OSX monitoring projects. Here are a few that came up on the radar.

  • Frameworks
    • Google Rapid Response - an incident response framework focused on remote live forensics.
    • Mig - a platform to perform investigative surgery on remote endpoints.
    • Osquery - an operating system instrumentation framework for OS X and Linux.
    • Blitz4j - a logging framework built on top of log4j to reduce contention and enable highly scalable logging without affecting application performance characteristics.
  • File Event Monitors
    • fswatch - cross-platform file change monitor with multiple backends.
      • FSW was created to replace fswatch but then they merged and fswatch became primary, again.
    • fslogger - an update of the original fslogger
      • Requires OpenSource XNU to compile. No need to build, just run the compilation line in the build instructions.

I quickly realized that I will need to spend a lot more time investigating, configuring, and testing the three incident response frameworks. What I was actually looking for appears to be handled by the file event monitors. These projects leverage Apple’s FSEvent mechanism. FSEvents are used by programs such as Spotlight and TimeMachine to monitor for changes in directories. Amit Singh outlines this much better than I ever could. I recommend you take a few moments to review his comments on FSEvent and how programs impact and are impacted by the normal operating process.

To start testing I installed both fswatch and fslogger to a clean installation of OSX. I did have to add a few programs, such as Homebrew, but I did my best to keep the operating system as original as possible.

FSWatch Test

I initially tested fswatch as it appeared to be the most up-to-date and supported project. For a quick test I decided to generate a new file, edit it, and dump it to terminal. Here are the results.

  • Start fswatch to monitor the current user’s directory.
fswatch start
1
MBP:fswatch testadmin$ fswatch /Users/testadmin/
  • Touch and edit a file then try to update this webpage
fswatch test file changes
1
2
3
4
MBP:Downloads testadmin$ touch test_fswatch_touch.txt
MBP:Downloads testadmin$ vim test_fswatch_touch.txt
MBP:Downloads testadmin$ cat test_fswatch_touch.txt
here is some data
  • STDOUT from fswatch
fswatch output
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
MBP:fswatch testadmin$ fswatch /Users/testadmin/
/Users/testadmin/Downloads/test_fswatch_touch.txt
/Users/testadmin/Library/Saved Application State/com.googlecode.iterm2.savedState/windows.plist
/Users/testadmin/Library/Saved Application State/com.googlecode.iterm2.savedState/window_2.data
/Users/testadmin/Library/Caches/com.apple.QuickLookDaemon/Cache.db-shm
/Users/testadmin/Library/Application Support/Quick Look/cloudthumbnails.db-shm
/Users/testadmin/Downloads/.test_fswatch_touch.txt.swx
/Users/testadmin/Downloads/.test_fswatch_touch.txt.swp
/Users/testadmin/Downloads/4913
/Users/testadmin/Downloads/test_fswatch_touch.txt~
/Users/testadmin/.viminfo.tmp
/Users/testadmin/.viminfo
/Users/testadmin/Downloads/.test_fswatch_touch.txt.swp
/Users/testadmin/Downloads/test_fswatch_touch.txt
[snip for brevity]
^CMBP:fswatch testadmin$

This output is very clean and concise. It provides information about files that have been updated but it does not provide any data about what happened to the file. Was it accessed? Was it modified? Was it created?

FSLogger Test

Next I tested fslogger. This program, as detailed in the source code, has to be compiled using Apple’s OpenSource XNU libraries. Running the same test, fslogger provided the following information about file changes.

fslogger output
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
MBP:fslogger testadmin$ sudo ./fslogger
fsevents device cloned (fd 4)
fslogger ready
[snip for brevity]
=> received 116 bytes
# Event
  type           = FSE_CREATE_FILE
  pid            = 29541 (exited?)
  # Details
    # type           len  data
    FSE_ARG_STRING    50  string = /Users/testadmin/Documents/test_fswatch_touch.txt
    FSE_ARG_DEV        4  dev    = 0x040007 (major 0, minor 262151)
    FSE_ARG_INO        4  ino    = 262154
    FSE_ARG_MODE       4  mode   = ?-----x---  (0x040008, vnode type VNON)
    FSE_ARG_UID        4  uid    = 262155 (?)
    FSE_ARG_GID        4  gid    = 524293 (?)
    FSE_ARG_INT64      8  tstamp = 2167408001296608
    FSE_ARG_DONE (0xb33f)
=> received 2619 bytes
# Event
  type           = FSE_CREATE_FILE
  pid            = 99 (cfprefsd)
  # Details
    # type           len  data
    FSE_ARG_STRING    75  string = /private/var/root/Library/Preferences/com.apple.cache_delete.plist.luwVtoi
    FSE_ARG_DEV        4  dev    = 0x040007 (major 0, minor 262151)
    FSE_ARG_INO        4  ino    = 262154
    FSE_ARG_MODE       4  mode   = ?-----x---  (0x040008, vnode type VNON)
    FSE_ARG_UID        4  uid    = 262155 (?)
    FSE_ARG_GID        4  gid    = 524293 (?)
    FSE_ARG_INT64      8  tstamp = 2448882978007264
    FSE_ARG_DONE (0xb33f)
[snip for brevity]

The difference in output is obvious. It seems that fslogger outputs information about what actually happened to the file. This output can be copied to a file and reviewed for IOCs.

Birth of fslogger-yaml

Hopefully you have already noticed the issue with IOC review from data provided by fslogger. The output from fslogger is just text. It is not delimited in a fashion that would benefit easy processing. Nobody wants to generate a list of IOCs by hand and I am no exception. Initially I started thinking about how I would approach processing this output with a Python script. While not specifically delimited there are several things that could be used to process the file in a logical manner. But this seemed tedious and time consuming.

I started looking at the source code for fslogger and I determined that it was simple enough that even “I” could update the C code. I devised a plan to insert delimiters into the output that would make processing via a Python script easy. Fortunately, I was actively talking to John H. Sawyer about this testing and my code updates. His comment to me was “Why don’t you output it in YAML?” At first I thought “isn’t that for configuration files” but then I shrugged and looked it up. Sure enough, YAML uses a simple syntax that would only require a few modifications to the fslogger source code. A comment marker (#) here, a space there, and the data captured by fslogger could be output in a syntax that would allow parsing scripts in any language. Yup, that John is a smart guy.

After some trial and error the YAML updates were injected into the fslogger source code. The following data shows the output for fslogger-yaml using the same test steps. In this case, the output demonstrates the fact that the test file was renamed and deleted. For those unfamiliar with YAML, the YAML formatting is hard to differentiate from the original fslogger output. After you research YAML a bit more you will notice that the status messages have been commented, each event is separated by a “—” document delimiter, and the white space (your favorite delimiter Tom) at the beginning of each line provides the necessary delimitation to map items.

fslogger-yaml output
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
#fsevents device cloned (fd 5)
#fslogger ready
# => received 614 bytes
---
[snip for brevity]
---
Event:
 type: FSE_RENAME
 pid: 30084
 pname: exited?
Details:
 FSE_ARG_STRING:
   len: 51
   string: /Users/testadmin/Documents/test_fslogger_touch.txt
 FSE_ARG_DEV:
   len: 4
   dev: 0x040007 (major 0, minor 262151)
 FSE_ARG_INO:
   len: 4
   ino: 262154
 FSE_ARG_MODE:
   len: 4
   mode: ?-----x---  (0x040008, vnode type VNON)
 FSE_ARG_UID:
   len: 4
   uid: 262155 (?)
 FSE_ARG_GID:
   len: 4
   gid: 3866626 (?)
 FSE_ARG_STRING:
   len: 59
   string: /Users/testadmin/Documents/test_fslogger_touch_renamed.txt
 FSE_ARG_INT64:
   len: 8
   tstamp: 197083164324371
 FSE_ARG_DONE:
   len: 0
   type: 45887
# => received 125 bytes
---
Event:
 type: FSE_DELETE
 pid: 30088
 pname: exited?
Details:
 FSE_ARG_STRING:
   len: 59
   string: /Users/testadmin/Documents/test_fslogger_touch_renamed.txt
 FSE_ARG_DEV:
   len: 4
   dev: 0x040007 (major 0, minor 262151)
 FSE_ARG_INO:
   len: 4
   ino: 262154
 FSE_ARG_MODE:
   len: 4
   mode: ?-----x---  (0x040008, vnode type VNON)
 FSE_ARG_UID:
   len: 4
   uid: 262155 (?)
 FSE_ARG_GID:
   len: 4
   gid: 524293 (?)
 FSE_ARG_INT64:
   len: 8
   tstamp: 7165142615334072854
 FSE_ARG_DONE:
   len: 0
   type: 45887
# => received 1106 bytes
[snip for brevity]

Thus, fslogger-yaml is born. The project includes a Python parser script. This script is just a proof-of-concept and I’m sure most of you, that have made it this far, will understand it easily. I’m hoping that John will add a Ruby parser to the project to help demonstrate the usefulness of outputing this data in YAML format. Ultimately I am hoping that this update will help automate the generation of IOCs and assist organizations with their incident response efforts.

If you have any questions or issues with fslogger-yaml then let me know via the GitHub project. That way we can track updates.

Go forth and do good things,
Don C. Weber (cutaway)