Tuesday, October 29, 2013

Exceptions.plist: interesting tweaks, including Gatekeeper forced opt-in

The Exceptions.plist is quite interesting:
defaults read /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist
Apple is using it to force opt-in a variety of apps for Gatekeeper protection by setting "LSFileQuarantineEnabled = 1". The list of apps on mavericks as of today is:
Apple also applies uses Exceptions.plist to apply categories to apps that aren't distributed through the appstore (where categories are presumably applied to all apps) e.g.
        "com.Google.GoogleEarthPlus" =         {
            LSApplicationCategoryType = "public.app-category.reference";
There's also LSMaximumVersionRequiringJava, which I think probably controls whether the OS should prompt the user to install Java for particular versions of various software:
        "com.adobe.Photoshop" =         {
            Java =             {
            LSApplicationCategoryType = "public.app-category.graphics-design";
            LSMaximumVersionRequiringJava = "12.19";
And miscellaneous tweaks, these for power management and display:
    AppNapOverrides =     {
        "com.apple.CrashTestDummy" =         (
                SoftDisabled = 1;
    HighResolutionOverrides =     {
        "com.apple.AppleQmaster" =         (
                HighVersion = "3.5.3";
                SoftDisabled = 1;
Daniel's documented VMWare version blacklisting for 10.8 appears to be gone in 10.9.

BIOS malware

Interesting recent developments on the BIOS malware front. MITRE delivered Copernicus and this blackhat talk, and Dragos Ruiu is on the tail of something very interesting if the claims can be believed and it isn't just a long troll...

Update: picked up by arstechnica.

Saturday, October 26, 2013

OS X XProtect

XProtect is Apple’s built-in AV system for OS X. I took at look at it on 10.8 (Mountain Lion). Virus definitions are stored in a simple format here:
This is part of the definition for a variant of MacDefender:
It contains a simple cleartext signature for a binary string in the Info.plist inside the MacDefender installer:
In [1]: a="4946506B67466C6167417574686F72697A6174696F6E416374696F6E3C2F6B65793E0A093C737472696E673E4E6F417574686F72697A6174696F6E3C2F737472696E673E0A093C6B65793E4946506B67466C616744656661756C744C6F636174696F6E3C2F6B65793E0A093C737472696E673E2F4170706C69636174696F6E733C2F737472696E673E0A09"

In [2]: a.decode('hex')
The XProtect updater (/usr/libexec/XProtectUpdater) runs as a launch daemon (/System/Library/LaunchDaemons/com.apple.xprotectupdater.plist), set to run every 24 hours and at load time by default. XProtectUpdater works by downloading a signed version of the plist from apple.

It (I assume) verifies the signature, and checks the version number in the downloaded version:
    <key>com.macromedia.Flash Player.plugin</key>
against the one recorded in:
and updates the metadata and signature plists if the version on disk is older. An sqlite database seems to be used as part of the update process:

Observing detection/cleaning behaviour

When I first started looking at XProtect Apple didn't have signatures for eicar, so seeing it in action was a little tricky, or required real malware. Now they do and it looks like this:
Breaking that down:
  • Description will show up in the "will damage your computer" warning box as seen below.
  • Matches all criteria must match for alert to be shown
  • Identity a base64 encoded sha1 hash.
  • NSURLNameKey filename
  • MatchType whether to match the filetype. Values are MatchAny, or Match. In this case it relies purely on the extension part of the filename specified in NSURNNameKey (so eicar.com will match, but downloading the same file as eicar.txt won't, and downloading eicar.com and renaming it eicar.txt also will avoid the warning). This makes sense since com has no header and eicar is just a test AV file.
Here's a python snippet to show the identity string is the sha1 hash:
In [12]: identity = "M5WFbOgfK3OC3ucmAveYtkLxQUA="

In [13]: identity.decode("base64").encode("hex")
Out[13]: '3395856ce81f2b7382dee72602f798b642f14140'

In [14]: import hashlib

In [15]: hashlib.sha1(open("eicar.com").read()).hexdigest()
Out[15]: '3395856ce81f2b7382dee72602f798b642f14140'
The signature format also supports various other NSURL filesystem resource keys as seen in this sig:
Lets look at a full signature, OpinionSpy is the simplest, here it is in full:
Note that type match is specified by UTI, and we have a pattern match rather than just a hash match. Replacing {4} and {12} with 4 and 12 bytes of stuff (I chose A's), here is a one-liner that generates a file that matches the signature:
$ python -c "open('eicar.jar','w').write('504B010214000A0000000800547D8B3B9B0231BCAAAAAAAA502D0700250000000000AAAAAAAAAAAAAAAAAAAAAAAA636F6D2F697A666F7267652F697A7061636B2F70616E656C732F706F696E7374616C6C6572'.decode('hex'))"
Which gives you part of a jar file (this signature works against the jar archive itself, not the contents):
$ hexdump -C eicar.jar 
00000000  50 4b 01 02 14 00 0a 00  00 00 08 00 54 7d 8b 3b  |PK..........T}.;|
00000010  9b 02 31 bc aa aa aa aa  50 2d 07 00 25 00 00 00  |..1.....P-..%...|
00000020  00 00 aa aa aa aa aa aa  aa aa aa aa aa aa 63 6f  |..............co|
00000030  6d 2f 69 7a 66 6f 72 67  65 2f 69 7a 70 61 63 6b  |m/izforge/izpack|
00000040  2f 70 61 6e 65 6c 73 2f  70 6f 69 6e 73 74 61 6c  |/panels/poinstal|
00000050  6c 65 72                                          |ler|
If you double-click the file, you'll see something like this since the jar doesn't actually work:

XProtect doesn't detect it since the quarantine bit isn't present. Upload it somewhere and download it with a browser to get the quarantine bit applied. Double-click and you will see this warning:

If you leave 'report malware' ticked, regardless of whether you click OK or Cancel, the malware report is sent to Apple and you get a nice malware report in:
which looks like:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
and similar information is written to system.log:
Sep 26 10:28:29 my-MacBook-Pro.local CoreServicesUIAgent[4961]: Adding LSProperties {
            LSQuarantineAgentBundleIdentifier = "com.google.Chrome";
            LSQuarantineAgentName = "Google Chrome";            
            LSQuarantineDataURL = "https://some.download.url";
            LSQuarantineEventIdentifier = "EEEEEEEE-DEAD-DEAD-DEAD-BEEFBEEFBEEF";
            LSQuarantineIsOwnedByCurrentUser = 1;
            LSQuarantineOriginURL = "https://some.referrer/url";
            LSQuarantineTimeStamp = "2012-09-26 17:09:40 +0000";
            LSQuarantineType = LSQuarantineTypeWebDownload;
        } to malware report for file://localhost/Users/me/Downloads/eicar.jar
Sep 26 10:28:29 my-MacBook-Pro.local CoreServicesUIAgent[4961]: Saved diag report for XProtect version ??? to /Users/me/Library/Logs/DiagnosticReports/XProtect_2012-09-26-102829_my-MacBook-Pro.diag
Sep 26 10:28:29 my-MacBook-Pro.local CoreServicesUIAgent[4961]: Successfully submitted CR malware report
Sep 26 10:28:29 my-MacBook-Pro com.apple.launchd.peruser.501[126] ([0x0-0x182182].com.apple.JarLauncher[4960]): Exited: Killed: 9
If you untick 'report malware', no report is written and nothing about the malware is logged, which is terrible for sysadmins who want to know what malware has been found on their machines :(

If you try running the same file (with quarantine) from the commandline, XProtect won't fire because it only applies to files launched through LaunchServices, similar to GateKeeper.
$ java -jar eicar.jar 
Invalid or corrupt jarfile eicar.jar

Modifying signatures

Modifying signatures is as simple as changing the plist:
I tested by downloading eicar as above, checking it was detected, then modifying the signature in the XProtect.plist and re-launching: it wasn't detected. When I put the original signature back, it was detected once again.

I didn't test adding signatures, but since the plist seems to be unprotected from modification (i.e. the signature checked at download time isn't stored and checked when it is read from disk), and changes take effect immediately, it should be as simple as modifying the plist. After testing with XprotectUpdater, local modifications seem to persist as long as the version number is higher in the local version.


  • Signatures are in the clear. An attacker who has been signatured doesn't need to spend any time figuring out what is signatured and testing modifications, they can just release a new version that evades the signature. Not a big deal since this is usually fairly easy without access to the actual signature, and since XProtect only has a small number of infrequently updated signatures the attacker will likely have to adapt to beat 3rd-party AV long before this anyway.
  • Only covers files with the quarantine bit set that are run via LaunchServices. Malware delivered by binaries that haven't opted-in with LSFileQuarantineEnabled won't be detected, and malware that writes files outside the normal APIs (e.g. with a Java browser plugin exploit) probably also won't be detected. Execution via the commandine or POSIX APIs also won't be blocked. Apple is really only targeting the download and double click malware that has been common to-date.
  • XProtect relies on unsigned plists for storing signatures and for storing XProtectUpdater version information that controls updates.  However, while most mac users continue to run as admin malware doesn't need to target these, it can overwrite the updater itself: one variant of Flashback already did this, which is presumably why the MRT was written and distributed via software update.

Update for Mavericks (10.9)

Some things have changed in 10.9. XProtectUpdater seems to have been removed or subsumed into another part of the OS, possibly part of the regular software update. This seems like a good move considering the comment above about malware attacking it. If I find out more I'll write another post.

Friday, October 25, 2013

Python: object container's __str__ calls __repr__ for the objects it contains

This should look familiar:
In [6]: class NewShiny(object):
    def __str__(self):
        return "something readable"

In [7]: a = NewShiny()

In [8]: print a
something readable
But have you ever been surprised by this?
In [10]: print [a]
[<__main__.NewShiny object at 0x4ce1590>]
What's happening is that when you call __str__ on the array it is calling __repr__ on all the objects inside the array. So the solution is to define __repr__:
In [15]: class NewShiny(object):
   ....:     def __str__(self):
   ....:         return "something readable"
   ....:     def __repr__(self):
   ....:         return "use this to recreate with eval()"

In [16]: a = NewShiny()

In [21]: print (a)
something readable

In [22]: print [a]
[use this to recreate with eval()]

In [23]: print (a,a)
(use this to recreate with eval(), use this to recreate with eval())
This stackoverflow answer has an excellent explanation of why this is the standard behaviour, and what the difference is between __str__ and __repr__. The general idea with __repr__ is that it returns a string that can be used to re-create the object if passed to eval().

OS X file types: Uniform Type Identifiers (UTI)

As a follow on from the post about MIME types and default handlers on OS X, here's a bit more about types in general. Apple explains it created the Uniform Type Identifier System in part to allow developers to register for types using a tree hierarchy, so for example if you can handle a text file you can probably handle a lot of text-based filetypes (like XML, HTML etc.). It also abstracts away the underlying complexity of type identification e.g. (multiple extensions like .jpg, .jpeg, and MIME type of image/jpeg).

You declare new UTIs in your application Info.plist. Apple core types can be seen here:
e.g. jpeg looks like this:
            UTTypeConformsTo = "public.image";
            UTTypeDescription = "JPEG image";
            UTTypeIdentifier = "public.jpeg";
            UTTypeTagSpecification =             {
                "com.apple.ostype" = JPEG;
                "public.filename-extension" =                 (
                "public.mime-type" =                 (
and the same file also contains a MIME type to file extension map:
    MIMETypeToExtensionMap =     {
        "application/andrew-inset" =         (
        "application/mac-compactpro" =         (
        "application/msexcel" =         (
        "application/mspowerpoint" =         (
        "application/msword" =         (
        "application/octet-stream" =         (

Thursday, October 17, 2013

OpenBSM auditd on OS X: these are the logs you are looking for

OpenBSM is an implementation of the now decades old Sun Basic Security Module (BSM) API and file format, written for OS X Panther (10.3) by McAfee Research under contract to Apple. The work was completed to provide OS X with security auditing, a core component required for Common Criteria evaluation, which is itself a necessary step for doing business with the US Federal Government and other governments around the world that respect the standard.

Robert Watson, one of the original authors, gives some background on the project in this interview, including the rationale for choosing BSM:
During the work for Apple, we identified Sun's BSM API and file format as the de facto industry standard for UNIX audit implementations -- it was extensively documented, the foundation of a previously evaluated system, and was extensible to new events and data types.
Apple released the code under a FreeBSD license, enabling inclusion in FreeBSD, currently maintained by Trusted BSD.

Before 10.6 the common criteria audit tools needed to be downloaded and installed separately, but have been installed by default since then. The functionality is a little-known goldmine for security purposes. Der Flounder has a good introduction to the configuration, but I'm going to cover some different aspects.

The /etc/security/audit_control file specifies what to log and retention/rotation policies:
# $P4: //depot/projects/trustedbsd/openbsm/etc/audit_control#8 $
expire-after:60d AND 1G
  • dir: where to write the logs
  • minfree: minimum % disk free until /etc/security/audit_warn starts getting called. You can customise/replace this script. The default is simply:
    logger -p security.warning "audit warning: $@"
  • flags: what user-attributable audit event classes are recorded for all users. You'll want to define a custom class (zz), more on that in a moment.
  • naflags: what non-user-attributable (system daemons etc.) event classes are recorded for all users. This config records the same events as specified by zz.
  • policy: cnt allows processes to still run if they aren't being audited (e.g. if the disk is full), argv records commandline arguments to execve. The other options are mostly either crazy sauce (halting the system if unable to audit), or not implemented.
  • filesz: rotation size threshold
  • expire-after: conditions for expiration of log files (removal)
  • everything else: here be undocumented dragons.
The /etc/security/audit_class file defines some default groups of events, which you can use in flags/naflags above to decide which events to record. I found I needed to create a custom class to log the right types of events. Here's the default file with a custom class added:
# $P4: //depot/projects/trustedbsd/openbsm/etc/audit_class#6 $
# $FreeBSD$
0x00000000:no:invalid class
0x00000001:fr:file read
0x00000002:fw:file write
0x00000004:fa:file attribute access
0x00000008:fm:file attribute modify
0x00000010:fc:file create
0x00000020:fd:file delete
0x00000040:cl:file close
0x00000400:na:non attributable
0x00002000:aa:authentication and authorization
0xffffffff:all:all flags set
Then you need to annotate the events you want to log with your custom class in /etc/security/audit_event. An example line to log execve (process execution) is below:
There is a huge menu of events to choose from, here are some examples: network socket connect/bind/accept, clock changes, auditctl, login/logout, mount, reboot, passwd, crontab modification, ssh, create/modify/update user/group, execve, fork, chmod, chown and many more.

The final piece of configuration is /etc/security/audit_user, which allows for per-user customisation that is combined with the global settings. This allows you to, for example, treat root differently to other users. The format is
and the default, below, only contains an customisation for root. alwaysaudit is 'lo' which records login/logout, password changes etc. while neveraudit is 'no' which is no exclusions.
# $P4: //depot/projects/trustedbsd/openbsm/etc/audit_user#3 $
# $FreeBSD: head/contrib/openbsm/etc/audit_user 157137 2006-03-26 01:44:35Z rwatson $
The auditd LaunchDaemon (/System/Library/LaunchDaemons/com.apple.auditd.plist which runs /usr/sbin/auditd) is responsible for handling the logfiles, rotation etc. There is a commandline utility /usr/sbin/audit which can be used to start/stop/reconfigure the daemon. "audit -s" is supposed to cause a re-sync of configuration, but I found a reboot was often the only really reliable way of making sure new config was applied.

You can get a live feed of the events being logged using praudit, which is very handy for debugging and checking configuration:
sudo praudit -l /dev/auditpipe
This is also the tool to use to get a textual representation of the binary auditlog files which are in the format of starttime.endtime:
praudit -l /var/audit/20130921124047.20130924235958
There is a GUI app available in the app store for visualising the binary log files, I haven't used it, but it looks reasonable for inspecting small amounts of logs. I wanted to apply existing large-scale log analysis capabilities, which would have been easy had the logs been sent to Apple SysLog (ASL) in plaintext. Sadly they aren't, so if you want to use them off the host you're going to need to either use praudit to convert them to text and ship them, or pump them into ASL yourself, or ship the binary log and convert it to text on the server-side. Running praudit over a set of logs is quite CPU intensive.

auditreduce allows you to read binary audit files and output filtered binary files, but its capabilities are very limited. You can't filter by IP for example (although the functionality was obviously planned since it is partially implemented, it will take a -o sock="" value, but the filter doesn't get applied). This produces events related to any file under /etc:
auditreduce -o file="/etc" /var/audit/20131018010044.20131023043346 | praudit -l
An event output by praudit will look like this (the one I grabbed happened to be an auditd log rotation):
# auditreduce -m AUE_EXECVE /var/audit/20131023061325.20131023062325 | praudit -l
header,159,11,execve(2),0,Tue Oct 22 23:23:25 2013, + 131 msec,exec arg,/usr/sbin/audit,-n,path,/usr/sbin/audit,path,/usr/sbin/audit,attribute,100755,root,wheel,16777220,2190270,0,subject,-1,root,wheel,root,wheel,50716,100000,0,,return,success,0,trailer,159,
Since that is basically gibberish, you can use the praudit xml output to get a better idea of what each value means:
# auditreduce -m AUE_EXECVE /var/audit/20131023061325.20131023062325 | praudit -x
<record version="11" event="execve(2)" modifier="0" time="Tue Oct 22 23:23:25 2013" msec=" + 131 msec" >
<attribute mode="100755" uid="root" gid="wheel" fsid="16777220" nodeid="2190270" device="0" />
<subject audit-uid="-1" uid="root" gid="wheel" ruid="root" rgid="wheel" pid="50716" sid="100000" tid="0" />
<return errval="success" retval="0" />

Tuesday, October 8, 2013

Fix bad timezones in EXIF on JPEG photos

If you're like me you always forget to change the timezone on your camera when you fly somewhere. It's often not until partway through the trip that I remember. Unless you remember to change the timezone before you take any pictures it's probably easiest to leave the camera in your home timezone. In fact, this is better all round since then you don't need to remember to switch it back either.

If you feel compelled to change it, take a photo of your watch (or just your wrist since no-one wears a watch these days) as a marker to remind yourself at what point the times change, since you'll need to work on just the photos with the wrong zone. If you're taking photos in many different timezones that's probably a good idea too.

When you get your photos to a computer there's a quick easy way to fix the time information on your photos on linux. Use exiv2 to modify the EXIF timestamp, this shifts the timestamps back by three hours:
exiv2 -a -03 adjust *.JPG
Then use this to set all the file modification times to the same value as the EXIF:
exiv2 -T rename *.JPG