Thursday, November 17, 2011

Create an array of identical strings in python

To create an array of identical strings in python (sometimes useful for testing):
In [12]: list("sdfsd" for x in range(1,4))
Out[12]: ['sdfsd', 'sdfsd', 'sdfsd']

Update: a commenter points out a better way. I knew you could do
but didn't think to try this. Thanks!

A shorter version:
In [16]: ["sdfsd"]*5
Out[16]: ['sdfsd', 'sdfsd', 'sdfsd', 'sdfsd', 'sdfsd']

And it's faster too....
$ python -mtimeit -n10000 "list('sdfsd' for i in xrange(1000))"
10000 loops, best of 3: 69.7 usec per loop
$ python -mtimeit -n10000 "['sdfsd']*1000"
10000 loops, best of 3: 5.66 usec per loop

Wednesday, November 16, 2011

OS X groups, subgroups and GeneratedUIDs

OS X groups are more confusing than they need to be, due to the use of UUIDs. First, lets create a group to test with and include a single user (test1):
$ sudo dseditgroup -o create -t group test00
$ sudo dseditgroup -o edit -a test1 -t user test00
$ dseditgroup -o read test00
dsAttrTypeStandard:GeneratedUID -
dsAttrTypeStandard:RecordName -
dsAttrTypeStandard:AppleMetaNodeLocation -
dsAttrTypeStandard:GroupMembers -
dsAttrTypeStandard:RecordType -
dsAttrTypeStandard:NestedGroups -
dsAttrTypeStandard:PrimaryGroupID -
dsAttrTypeStandard:GroupMembership -
Note that we have GroupMembership that includes test1 and the GroupMembers attribute lists the GUID for the test1 user. Neither of these attributes tell us about groups that are members of this group. To see how they are stored, we will add a subgroup:
$ sudo dseditgroup -o edit -a testsubgroup -t group test00
$ dseditgroup -o read test00
dsAttrTypeStandard:GeneratedUID -
dsAttrTypeStandard:RecordName -
dsAttrTypeStandard:AppleMetaNodeLocation -
dsAttrTypeStandard:GroupMembers -
dsAttrTypeStandard:RecordType -
dsAttrTypeStandard:NestedGroups -
dsAttrTypeStandard:PrimaryGroupID -
dsAttrTypeStandard:GroupMembership -
Note that the subgroup UUID is now listed under NestedGroups, but the human-readable name is not, presumably because apple wanted to make this as painful as possible. There isn't a nice way to map UUID to a recordname, but searching in the local node does work (although it seems to ignore the output formatting options):
$ dscl -plist . -search /Groups GeneratedUID 6B3D7A4A-A795-4569-8640-2105F73EEA6B
testsubgroup  GeneratedUID = (
So the outcome of this is that to code to answer a simple question: 'who are the members of this group?' is actually really complicated.

There is however a way to check if a user is a member of a group, and it will follow nested group relationships:
$ sudo dseditgroup -o edit -a test2 -t user testsubgroup
$ dseditgroup -o checkmember -m test2 test00
yes test2 is a member of test00

Tuesday, November 15, 2011

Using netgroups to control login and admin privileges on OS X

I wanted to use netgroups defined in /etc/netgroup to control:
  • Login access via access_loginwindow
  • OS X admin privs via the admin local group
  • sudo access via /etc/sudoers
Unfortunately I couldn't get it to work (on 10.6), but I'll document here for my own sake and the rest of the internet :) I suspect this was because I wanted to just use the flat file, without any NIS server, and OS X was expecting to do lookups to refresh the information in the file.

Run Directory Utility
open /System/Library/CoreServices/Directory\
and tick 'BSD Flat File and NIS' and inside there tick 'Use User and Group records in BSD Local node'. Interestingly this is gone on Lion - it is now just NIS and there is no option for BSD local node...

Assuming you have some entries in /etc/netgroup:
smalltest (-,auser,)
You should be able to see them with dscl:
$ dscl localhost -read /BSD/local/NetGroups/smalltest
dsAttrTypeNative:triplet: -,auser,
AppleMetaNodeLocation: /BSD/local
RecordName: smalltest
And should theoretically be able to use them. I tried this for sudo, and although it passed syntax checking it just didn't work.
+smalltest ALL=(ALL) ALL
And I couldn't get dseditgroup to recognise the netgroup for use in access_loginwindow:
$ sudo dseditgroup -o edit -a smalltest -t group
Group not found.

OS X Service ACLs*

To place ACLs on OS X services, you can create groups like this:
sudo dseditgroup -o create -t group
sudo dseditgroup -o edit -a auser -t user
which in this case will limit who can login at the console (i.e. loginwindow) to just the user 'auser'. You can check the membership with:
$ dscl . -read /Groups/
AppleMetaNodeLocation: /Local/Default
GroupMembership: auser
PrimaryGroupID: 504
RecordType: dsRecTypeStandard:Groups

Similar groups will control access to other services such as,

Monday, November 14, 2011

Compiling C libraries for use in python modules on OS X: specify a target architecture

My pycurl installation was failing with these errors:

$pip install pycurl


assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed

Installed assemblers are:

/usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64

/usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
The solution was to specify an architecture like this:
env ARCHFLAGS="-arch x86_64" pip install pycurl

Thursday, November 10, 2011

OS X logging: asl.conf, syslog.conf, syslog master filters and log rotation

The apple syslog daemon (/System/Library/LaunchDaemons/ takes configuration from 3 different sources *groan*.


asl.conf (the apple syslog configuration file) controls what data is stored in the apple syslog binary databases under /private/var/log/asl. It is powerful and allows everything from filtering:
# save everything from emergency to notice
? [<= Level notice] store

to access control. This restricts read access to uid 0 (root) and gid 80 (admin):
# authpriv messages are root/admin readable
? [= Facility authpriv] access 0 80


In addition, and independent of, asl.conf, syslogd also reads syslog.conf which is the familiar unix/linux/bsd style syslog configuration file. This allows you to write the same log lines to plaintext log files in /var/log (or wherever) that also get stored by asl in binary form as above.

syslogd is responsible for writing both the asl and regular /var/log files, which you can see with lsof:
$ sudo lsof | grep syslog
syslogd   65475           root    9w      REG                1,4   13178158  344319 /private/var/log/system.log
syslogd   65475           root   10w      REG                1,4       5884 3055612 /private/var/log/secure.log
syslogd   65475           root   11w      REG                1,4    1093081 3029374 /private/var/log/debug.log
syslogd   65475           root   12u      REG                1,4     170227 3024189 /private/var/log/asl/2013.01.03.U0.G80.asl
syslogd   65475           root   13u      REG                1,4    2204298 3024190 /private/var/log/asl/2013.01.03.G80.asl
You can see that syslogd writes into /private/var/log/system.log, which is hard-linked to /var/log/system.log (same inode):
$ ls -lai /var/log/system.log 
344319 -rw-r--r--+ 1 root  admin  13192052 Jan  3 14:29 /var/log/system.log
$ ls -lai /private/var/log/system.log 
344319 -rw-r--r--+ 1 root  admin  13192052 Jan  3 14:29 /private/var/log/system.log

Syslog master filters

As if that wasn't enough, the syslog daemon itself also has a global master filter rule, which you can inspect with:

$ syslog -c 0
Master filter mask: Debug

AND you can set per-process filters:
$ syslog -c syslogd
ASL Data Store filter mask: Emergency - Debug

Reading logs

To read asl logs, just use the syslog command. It actually comes with some nifty filtering of its own. For example this command shows log lines sent by login for the past 2 hours:
syslog -k Sender login -k Time ge -2h

To help you formulate queries like the above you can see the raw key value pairs using:
syslog -F raw

There is a gotcha in the default apple configuration which has lines like:
# redirect to /var/log/DiagnosticMessages
? [T] store_dir /var/log/DiagnosticMessages

# redirect* messages to /var/log/performance
? [A= Facility] store_dir /var/log/performance

# redirect* messages to /var/log/eventmonitor
? [A= Facility] store_dir /var/log/eventmonitor

Which means just using the syslog command as above doesn't give you all the logs, including stuff like apple updates:
$ syslog  | grep 'downloading "Thunderbolt Software Update, 1.0"'
To look at the DiagnosticMessages log use:
$ syslog -d /var/log/DiagnosticMessages/ | grep 'downloading "Thunderbolt Software Update, 1.0"'
Software Update[1865] : SWU: downloading "Thunderbolt Software Update, 1.0"

To read the BSD logs just go look at the files in /var/log.

Log Rotation

Regular (non-asl binary) log rotation configuration is in
and has configuration lines that look like this (ie. completely different to linux /etc/logrotate.d):
# logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
/var/log/appfirewall.log  640  5     1000 *     J

See the newsyslog.conf man page for the details.

ASL log rotation is handled by aslmanager, and configured with directives in asl.conf. See asl.conf and aslmanager man pages.

Tuesday, November 8, 2011

OS X Terminal screwed up command history in iPython

I was getting some weird behaviour in ipython (0.11) in a OS X (10.6.8) Terminal window. When I used the up arrow to access the commandline history, all long lines (and even some short lines) were screwed up - the text wrapping was wrong. Others have noticed the same thing and it turns out the solution is to re-install readline (which requires Xcode to compile it):

sudo pip install readline
Turns out on Lion with Xcode 4.2, the package downloads and installs without errors but doesn't actually fix the problem. If you download and install using the egg (of the same version?! 6.2.2) it works fine.
sudo easy_install readline-6.2.2-py2.7-macosx-10.7-intel.egg
Update: Just noticed this message on starting ipython on Mountain Lion, and 'sudo easy_install readline' works.
libedit detected - readline will not be well behaved, including but not limited to:
   * crashes on tab completion
   * incorrect history navigation
   * corrupting long-lines
   * failure to wrap or indent lines properly
It is highly recommended that you install readline, which is easy_installable:
     easy_install readline
Note that `pip install readline` generally DOES NOT WORK, because
it installs to site-packages, which come *after* lib-dynload in sys.path,
where readline is located.  It must be `easy_install readline`, or to a custom
location on your PYTHONPATH (even --user comes after lib-dyload).

Monday, November 7, 2011

Equivalent of 'watch' command for OS X

'watch' doesn't exist inbuilt for OS X like it does for linux. Someone has actually implemented it and it is available from fink, but here is a bash function that uses a simple while loop to do basically the same thing. Drop this into bash_aliases and you now have a simple version of watch.

watch () {
  while [ 1 ]; do "${@}" ; sleep 2 ; done

HOWTO write an OS X (seatbelt) sandbox profile

To create a simple seatbelt sandbox profile, start with a trace by saving this as

(version 1)
(trace "/tmp/")
sandbox-exec -f binary_to_be_sandboxed
sandbox-simplify /tmp/ > ./
Simplify crunches down the verbose log into a more compact profile. Edit it (you especially want to remove any spurious dtrace lines that are artifacts of the capture process) and run the binary in its sandbox:
sandbox-exec -f binary_to_be_sandboxed
This blog has a simple shell script to automate the above process, but you probably want to manually inspect and edit your profile before using it for real.

You can see a bunch of built-in sandboxes in
and you can view which running processes are sandboxed by adding the sandbox column to the activity monitor gui.

OS X kernel panic: "You need to restart your computer"

This is the OS X kernel panic screen:

Apple has some information on how you can configure OS X to send kernel core dumps over TCP and some information on how to debug panics. You can also find a panic log at