Thursday, December 29, 2011

OS X hidden folders, extended attributes, flags, and file ACLs

On lion the ~/Library directory is hidden by default in Finder. This doesn't bother me normally, since I rarely use Finder and it is visible on the commandline, but I recently got stuck needing to submit logs from ~/Library using a web-based form.

There are some one-off tricks you can use to access it, but, as far as I can see, no equivalent of the windows 'show hidden files' setting in Finder.

To see extended attributes (indicated by trailing @), file ACLs (indicated by trailing +), and flags, use this:
$ ls -l@eOd Library/
drwx------@ 38 user  group  hidden 1292 Dec 28 16:02 Library/
 com.apple.FinderInfo   32 
 0: group:everyone deny delete

You can manipulate the 'hidden' flag (which seems to actually be implemented as a FinderInfo attribute) and others with chflags:
chflags nohidden Library

Which gives:
$ ls -l@eOd Library/
drwx------+ 38 user  group  - 1292 Dec 28 16:02 Library/
 0: group:everyone deny delete

Tuesday, December 20, 2011

Simple cocoa dialogs for commandline tools in OS X

Got a commandline tool that needs to interact with a user? cocoaDialog is an app that can provide simple, good looking dialogs and can be invoked just by passing commandline options.

Sunday, December 18, 2011

NSA response to USB-borne malware

The Washington Post has an interesting article on the NSA's response to a USB-based malware infection on the US secret network in October 2008.

Wednesday, December 14, 2011

Ruby %x, system(), `backtick`, and puppet Facter::Util::Resolution.exec differences

Below are three different ways of running a system command from ruby:
irb(main):014:0> %x(echo "sdfsd")
=> "sdfsd\n"
irb(main):015:0> system('echo "sdfsd"')
sdfsd
=> true
irb(main):016:0> `echo "sdfsd"`
=> "sdfsd\n"
system() gives you a shell, but doesn't save the output. Backticks and %x are equivalent: they give you a shell environment and return the stdout output.

In puppet if you're using Resolution.exec:
Facter::Util::Resolution.exec('echo "sdfsd"')
Things are not so obvious. Depending on the puppet version, you might get a shell environment and puppet is also doing some checking with 'which' that might result in a 'nil' return if your command is not a simple binary.

Friday, December 9, 2011

Convert group and user GeneratedUIDs to human-readable form on OS X

When working with users and groups on OS X you frequently find that you need to lookup GUIDs to get their human-readable names. Here are a few examples of the fairly painful syntax (good candidates for bash aliases).

Get a group/user for a known GeneratedUID from local directory services:
dscl . -search /Users GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66
dscl . -search /Groups GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66

and from LDAP:
dscl /LDAPv3/ldap.company.com/ -search /Groups GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66
dscl /LDAPv3/ldap.company.com/ -search /Users GeneratedUID 00052AE8-5000-6000-9007-666F6B666A66
ldapsearch -LLLx -b ou=group,dc=company,dc=com "apple-generateduid=00052AE8-5000-6000-9007-666F6B666A66" cn
ldapsearch -LLLx -b ou=people,dc=company,dc=com "apple-generateduid=00052AE8-5000-6000-9007-666F6B666A66" uid

Get a GeneratedUID for a known user/group:
dscl localhost -read /Search/Users/auser | grep -m1 GeneratedUID | cut -c15-
dscl localhost -read /Search/Groups/agroup | grep -m1 GeneratedUID | cut -c15-
ldapsearch -LLLx -b ou=group,dc=company,dc=com "cn=agroup" apple-generateduid
ldapsearch -LLLx -b ou=people,dc=company,dc=com "uid=auser" apple-generateduid

Ruby Time and DateTime: parse a string, compute the difference, simple right?

Time in ruby is a confusing maze of suckiness.

The Ruby Time class doesn't include a method to convert a string into a Time object (like the C standard 'strptime'), despite having a Time.strftime function. What the?

It turns out that Time.parse and Time.strptime do exist, but confusingly they aren't in the core class, but in a separate library you have to require that overrides the class. WTF?

irb(main):001:0> Time.parse
NoMethodError: undefined method `parse' for Time:Class
 from (irb):1
irb(main):002:0> require 'time'
=> true
irb(main):003:0> Time.parse
ArgumentError: wrong number of arguments (0 for 1)
 from (irb):3:in `parse'
 from (irb):3

So it seems that the best way to read a time from a file and compare it to the current time is the following, which gives you an answer in seconds:
irb(main):010:0> a=Time.parse('2011-01-10 12:34:00 UTC')
=> Mon Jan 10 12:34:00 UTC 2011
irb(main):012:0> b=Time.now.utc
=> Fri Dec 09 20:47:32 UTC 2011
irb(main):013:0> b-a
=> 28800812.788154

The only downside of this is that the library never fails, it tries really hard to give you a Time object back, to the point where it will just return the current time if the string is rubbish:
irb(main):027:0> Time.parse('asdfasdfas')
=> Fri Dec 09 14:02:40 -0800 2011

whereas DateTime is actually more sensible:
irb(main):028:0> DateTime.parse('asdfasdfas')
ArgumentError: invalid date
 from /usr/lib/ruby/1.8/date.rb:1576:in `new_by_frags'
 from /usr/lib/ruby/1.8/date.rb:1621:in `parse'
 from (irb):28
 from  :0

So my final solution was:
  begin
    DateTime.parse(time_string)
  rescue ArgumentError
    return 'UNKNOWN'
  end
  earlier = Time.parse(time_string)

  now = Time.now.utc
  hours = (now-earlier)/3600
  return hours
Read on for an attempt to do the same with DateTime, which is a terrible, terrible API.

DateTime has a simple way to ingest strings (and also has a strptime):
irb(main):015:0> require 'date'
=> false
irb(main):023:0> a=DateTime.parse('2011-12-09 00:00:00 UTC')
=> #
irb(main):024:0> puts a
2011-12-09T00:00:00+00:00
=> nil

But you can't take a difference between Time and DateTime:
irb(main):027:0> a=DateTime.parse('2011-12-01 00:00:00 UTC')
=> #
irb(main):028:0> b=Time.now
=> Fri Dec 09 12:10:40 -0800 2011
irb(main):029:0> b-a
TypeError: can't convert DateTime into Float
 from (irb):29:in `-'
 from (irb):29
 from  :0

Subtracting two DateTimes works but gives you an answer that is a rational number of days (?!):
irb(main):031:0> b-a
=> Rational(76394798789, 8640000000)
irb(main):032:0> c=b-a
=> Rational(76394798789, 8640000000)
irb(main):033:0> puts c
76394798789/8640000000
=> nil
irb(main):034:0> puts c.to_f
8.8419906005787
=> nil

Since a fractional number of days is pretty useless you can then convert this to an array of [hours,minutes,seconds,frac]:
irb(main):036:0> DateTime.day_fraction_to_time(c)
=> [212, 12, 27, Rational(98789, 8640000000)]

Why the subtraction doesn't return another type of time object is beyond me.

Wednesday, December 7, 2011

auditd on OS X

Auditd gets a limited description in the Snow Leopard Security Config doc. I'm posting a quick summary here, and will update it as I learn more.

auditd rules are kept in /etc/security.  The audit_control rules apply to all users and audit_user allows for per-user rules.

Audit logs are stored in binary format in /var/audit/logstarttime.logfinishtime and can be read with:
praudit /var/audit/20111018000205.20111018000916

Thursday, November 17, 2011

Create an array of identical strings in python

To create an array of identical strings in python (sometimes useful for testing):
In [12]: list("sdfsd" for x in range(1,4))
Out[12]: ['sdfsd', 'sdfsd', 'sdfsd']

Update: a commenter points out a better way. I knew you could do
'a'*5
but didn't think to try this. Thanks!

A shorter version:
In [16]: ["sdfsd"]*5
Out[16]: ['sdfsd', 'sdfsd', 'sdfsd', 'sdfsd', 'sdfsd']

And it's faster too....
$ python -mtimeit -n10000 "list('sdfsd' for i in xrange(1000))"
10000 loops, best of 3: 69.7 usec per loop
$ python -mtimeit -n10000 "['sdfsd']*1000"
10000 loops, best of 3: 5.66 usec per loop

Wednesday, November 16, 2011

OS X groups, subgroups and GeneratedUIDs

OS X groups are more confusing than they need to be, due to the use of UUIDs. First, lets create a group to test with and include a single user (test1):
$ sudo dseditgroup -o create -t group test00
$ sudo dseditgroup -o edit -a test1 -t user test00
$ dseditgroup -o read test00
dsAttrTypeStandard:GeneratedUID -
  C110609F-0C0A-42E7-B663-D93BA03ECDAC
dsAttrTypeStandard:RecordName -
  test00
dsAttrTypeStandard:AppleMetaNodeLocation -
  /Local/Default
dsAttrTypeStandard:GroupMembers -
  6D456134-85E6-4912-88C7-7C6139057B9B
dsAttrTypeStandard:RecordType -
  dsRecTypeStandard:Groups
dsAttrTypeStandard:NestedGroups -
dsAttrTypeStandard:PrimaryGroupID -
  505
dsAttrTypeStandard:GroupMembership -
  test1
Note that we have GroupMembership that includes test1 and the GroupMembers attribute lists the GUID for the test1 user. Neither of these attributes tell us about groups that are members of this group. To see how they are stored, we will add a subgroup:
$ sudo dseditgroup -o edit -a testsubgroup -t group test00
$ dseditgroup -o read test00
dsAttrTypeStandard:GeneratedUID -
  C110609F-0C0A-42E7-B663-D93BA03ECDAC
dsAttrTypeStandard:RecordName -
  test00
dsAttrTypeStandard:AppleMetaNodeLocation -
  /Local/Default
dsAttrTypeStandard:GroupMembers -
  6D456134-85E6-4912-88C7-7C6139057B9B
dsAttrTypeStandard:RecordType -
  dsRecTypeStandard:Groups
dsAttrTypeStandard:NestedGroups -
  6B3D7A4A-A795-4569-8640-2105F73EEA6B
dsAttrTypeStandard:PrimaryGroupID -
  505
dsAttrTypeStandard:GroupMembership -
  test1
Note that the subgroup UUID is now listed under NestedGroups, but the human-readable name is not, presumably because apple wanted to make this as painful as possible. There isn't a nice way to map UUID to a recordname, but searching in the local node does work (although it seems to ignore the output formatting options):
$ dscl -plist . -search /Groups GeneratedUID 6B3D7A4A-A795-4569-8640-2105F73EEA6B
testsubgroup  GeneratedUID = (
    "6B3D7A4A-A795-4569-8640-2105F73EEA6B"
)
So the outcome of this is that to code to answer a simple question: 'who are the members of this group?' is actually really complicated.

There is however a way to check if a user is a member of a group, and it will follow nested group relationships:
$ sudo dseditgroup -o edit -a test2 -t user testsubgroup
$ dseditgroup -o checkmember -m test2 test00
yes test2 is a member of test00

Tuesday, November 15, 2011

Using netgroups to control login and admin privileges on OS X

I wanted to use netgroups defined in /etc/netgroup to control:
  • Login access via access_loginwindow
  • OS X admin privs via the admin local group
  • sudo access via /etc/sudoers
Unfortunately I couldn't get it to work (on 10.6), but I'll document here for my own sake and the rest of the internet :) I suspect this was because I wanted to just use the flat file, without any NIS server, and OS X was expecting to do lookups to refresh the information in the file.

Run Directory Utility
open /System/Library/CoreServices/Directory\ Utility.app/
and tick 'BSD Flat File and NIS' and inside there tick 'Use User and Group records in BSD Local node'. Interestingly this is gone on Lion - it is now just NIS and there is no option for BSD local node...

Assuming you have some entries in /etc/netgroup:
smalltest (-,auser,)
You should be able to see them with dscl:
$ dscl localhost -read /BSD/local/NetGroups/smalltest
dsAttrTypeNative:triplet: -,auser,
AppleMetaNodeLocation: /BSD/local
RecordName: smalltest
And should theoretically be able to use them. I tried this for sudo, and although it passed syntax checking it just didn't work.
+smalltest ALL=(ALL) ALL
And I couldn't get dseditgroup to recognise the netgroup for use in access_loginwindow:
$ sudo dseditgroup -o edit -a smalltest -t group com.apple.access_loginwindow
Group not found.

OS X Service ACLs com.apple.access_*

To place ACLs on OS X services, you can create groups like this:
sudo dseditgroup -o create -t group com.apple.access_loginwindow
sudo dseditgroup -o edit -a auser -t user com.apple.access_loginwindow
which in this case will limit who can login at the console (i.e. loginwindow) to just the user 'auser'. You can check the membership with:
$ dscl . -read /Groups/com.apple.access_loginwindow
AppleMetaNodeLocation: /Local/Default
GroupMembership: auser
PrimaryGroupID: 504
RecordName: com.apple.access_loginwindow
RecordType: dsRecTypeStandard:Groups

Similar groups will control access to other services such as com.apple.access_ssh, com.apple.access_screensharing.

Monday, November 14, 2011

Compiling C libraries for use in python modules on OS X: specify a target architecture

My pycurl installation was failing with these errors:

$pip install pycurl

[snip]

assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed

Installed assemblers are:

/usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64

/usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
The solution was to specify an architecture like this:
env ARCHFLAGS="-arch x86_64" pip install pycurl

Thursday, November 10, 2011

OS X logging: asl.conf, syslog.conf, syslog master filters and log rotation

The apple syslog daemon (/System/Library/LaunchDaemons/com.apple.syslogd.plist) takes configuration from 3 different sources *groan*.

asl.conf


asl.conf (the apple syslog configuration file) controls what data is stored in the apple syslog binary databases under /private/var/log/asl. It is powerful and allows everything from filtering:
# save everything from emergency to notice
? [<= Level notice] store

to access control. This restricts read access to uid 0 (root) and gid 80 (admin):
# authpriv messages are root/admin readable
? [= Facility authpriv] access 0 80

syslog.conf


In addition, and independent of, asl.conf, syslogd also reads syslog.conf which is the familiar unix/linux/bsd style syslog configuration file. This allows you to write the same log lines to plaintext log files in /var/log (or wherever) that also get stored by asl in binary form as above.

syslogd is responsible for writing both the asl and regular /var/log files, which you can see with lsof:
$ sudo lsof | grep syslog
[snip]
syslogd   65475           root    9w      REG                1,4   13178158  344319 /private/var/log/system.log
syslogd   65475           root   10w      REG                1,4       5884 3055612 /private/var/log/secure.log
syslogd   65475           root   11w      REG                1,4    1093081 3029374 /private/var/log/debug.log
syslogd   65475           root   12u      REG                1,4     170227 3024189 /private/var/log/asl/2013.01.03.U0.G80.asl
syslogd   65475           root   13u      REG                1,4    2204298 3024190 /private/var/log/asl/2013.01.03.G80.asl
You can see that syslogd writes into /private/var/log/system.log, which is hard-linked to /var/log/system.log (same inode):
$ ls -lai /var/log/system.log 
344319 -rw-r--r--+ 1 root  admin  13192052 Jan  3 14:29 /var/log/system.log
$ ls -lai /private/var/log/system.log 
344319 -rw-r--r--+ 1 root  admin  13192052 Jan  3 14:29 /private/var/log/system.log

Syslog master filters


As if that wasn't enough, the syslog daemon itself also has a global master filter rule, which you can inspect with:

$ syslog -c 0
Master filter mask: Debug

AND you can set per-process filters:
$ syslog -c syslogd
ASL Data Store filter mask: Emergency - Debug

Reading logs


To read asl logs, just use the syslog command. It actually comes with some nifty filtering of its own. For example this command shows log lines sent by login for the past 2 hours:
syslog -k Sender login -k Time ge -2h

To help you formulate queries like the above you can see the raw key value pairs using:
syslog -F raw

There is a gotcha in the default apple configuration which has lines like:
# redirect com.apple.message.domain to /var/log/DiagnosticMessages
? [T com.apple.message.domain] store_dir /var/log/DiagnosticMessages

# redirect com.apple.performance* messages to /var/log/performance
? [A= Facility com.apple.performance] store_dir /var/log/performance

# redirect com.apple.eventmonitor* messages to /var/log/eventmonitor
? [A= Facility com.apple.eventmonitor] store_dir /var/log/eventmonitor

Which means just using the syslog command as above doesn't give you all the logs, including stuff like apple updates:
$ syslog  | grep 'downloading "Thunderbolt Software Update, 1.0"'
To look at the DiagnosticMessages log use:
$ syslog -d /var/log/DiagnosticMessages/ | grep 'downloading "Thunderbolt Software Update, 1.0"'
Software Update[1865] : SWU: downloading "Thunderbolt Software Update, 1.0"

To read the BSD logs just go look at the files in /var/log.

Log Rotation


Regular (non-asl binary) log rotation configuration is in
/etc/newsyslog.conf
and has configuration lines that look like this (ie. completely different to linux /etc/logrotate.d):
# logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
/var/log/appfirewall.log  640  5     1000 *     J

See the newsyslog.conf man page for the details.

ASL log rotation is handled by aslmanager, and configured with directives in asl.conf. See asl.conf and aslmanager man pages.

Tuesday, November 8, 2011

OS X Terminal screwed up command history in iPython

I was getting some weird behaviour in ipython (0.11) in a OS X (10.6.8) Terminal window. When I used the up arrow to access the commandline history, all long lines (and even some short lines) were screwed up - the text wrapping was wrong. Others have noticed the same thing and it turns out the solution is to re-install readline (which requires Xcode to compile it):

sudo pip install readline
Turns out on Lion with Xcode 4.2, the package downloads and installs without errors but doesn't actually fix the problem. If you download and install using the egg (of the same version?! 6.2.2) it works fine.
sudo easy_install readline-6.2.2-py2.7-macosx-10.7-intel.egg
Update: Just noticed this message on starting ipython on Mountain Lion, and 'sudo easy_install readline' works.
******************************************************************************
libedit detected - readline will not be well behaved, including but not limited to:
   * crashes on tab completion
   * incorrect history navigation
   * corrupting long-lines
   * failure to wrap or indent lines properly
It is highly recommended that you install readline, which is easy_installable:
     easy_install readline
Note that `pip install readline` generally DOES NOT WORK, because
it installs to site-packages, which come *after* lib-dynload in sys.path,
where readline is located.  It must be `easy_install readline`, or to a custom
location on your PYTHONPATH (even --user comes after lib-dyload).
******************************************************************************

Monday, November 7, 2011

Equivalent of 'watch' command for OS X

'watch' doesn't exist inbuilt for OS X like it does for linux. Someone has actually implemented it and it is available from fink, but here is a bash function that uses a simple while loop to do basically the same thing. Drop this into bash_aliases and you now have a simple version of watch.

watch () {
  while [ 1 ]; do "${@}" ; sleep 2 ; done
}

HOWTO write an OS X (seatbelt) sandbox profile

To create a simple seatbelt sandbox profile, start with a trace by saving this as trace.sb:

(version 1)
(trace "/tmp/traceout.sb")
Then
sandbox-exec -f trace.sb binary_to_be_sandboxed
sandbox-simplify /tmp/traceout.sb > ./tracesimple.sb
Simplify crunches down the verbose log into a more compact profile. Edit it (you especially want to remove any spurious dtrace lines that are artifacts of the capture process) and run the binary in its sandbox:
sandbox-exec -f tracesimple.sb binary_to_be_sandboxed
This blog has a simple shell script to automate the above process, but you probably want to manually inspect and edit your profile before using it for real.

You can see a bunch of built-in sandboxes in
/usr/share/sandbox
and you can view which running processes are sandboxed by adding the sandbox column to the activity monitor gui.

OS X kernel panic: "You need to restart your computer"

This is the OS X kernel panic screen:

Apple has some information on how you can configure OS X to send kernel core dumps over TCP and some information on how to debug panics. You can also find a panic log at
/Library/Logs/DiagnosticReports/Kernel_time_host.panic

Friday, October 28, 2011

NFS shares

Show NFS shares exported from a host (can also point to localhost):
showmount -e 192.168.1.1
/etc/exports line on linux (share /blah with 192.168.1.2)
/blah               192.168.1.2(ro)
On the mac the options format is different. As with linux, root_squash to nobody is the default.
/blah -ro 192.168.1.2

Tuesday, October 25, 2011

Installing pip and virtualenv on OS X

Download and install the appropriate easy_install egg.
sudo sh setuptools-0.6c11-py2.X.egg
and from then on it is the same as for linux
sudo easy_install pip
sudo pip install virtualenv
Create a virtualenv instance:
virtualenv ENV
and you'll get something like:
Could not call install_name_tool -- you must have Apple's development tools installed
So you need to go off and install XCode, which will probably take a while (4.2 is 1.7 G).

Once you have Xcode, things should be good:
$ virtualenv ENV
New python executable in ENV/bin/python
Installing setuptools............done.
Installing pip...............done.
$ source ENV/bin/activate

Thursday, October 20, 2011

OSX launchd and inetd services

Apple has an epic writeup of how daemons should be written for OS X, Mach execution contexts, IPC and more in TN2083. This post is just some quick summary notes.

launchd is roughly equivalent to cron, init.d, and xinetd all rolled into one. Use it by dropping a plist into one of these directories:

  • /Library/LaunchDaemons runs as root on system startup
  • /Library/LaunchAgents runs as the user on login
  • ~/Library/LaunchAgents runs as the user for the specific user login

There is also /System/Library/Launch{Agents,Daemons} which is for the jobs installed by the OS. Use the directories above for any custom jobs.

List launchd jobs with
launchctl list
Start:
sudo launchctl load /Library/LaunchDaemons/com.myorg.somejob.plist
Stop:
sudo launchctl unload /Library/LaunchDaemons/com.myorg.somejob.plist

There are many different options for jobs, see the launchd.plist man page for a complete list. A simple inetd service might look something like this:
<dict>
 <key>Disabled</key>
 <false/>
 <key>Label</key>
 <string>com.myorg.somejob</string>
 <key>Program</key>
 <string>/usr/local/sbin/somejob</string>
 <key>Sockets</key>
 <dict>
  <key>Listeners</key>
  <dict>
   <key>SockServiceName</key>
   <string>9999</string>
  </dict>
 </dict>
 <key>inetdCompatibility</key>
 <dict>
  <key>Wait</key>
  <false/>
 </dict>
</dict>

/usr/local/sbin/somejob will be called whenever there is a connection on port 9999, and you can interact with the socket by reading and writing STDIN/STDOUT as normal for inetd.

Wednesday, October 19, 2011

Force MCX refresh - apply policy without a logout/reboot

If you are managing settings with MCX (e.g. like this), you want your MCX settings to apply as soon as they are pushed to the machine. Unfortunately this doesn't happen automatically. They get stored in the local directory services node, which you can see here:
dscl . -mcxread /Computers/local_computer
But they don't get propagated to the relevant plists under managed preferences until a logout or reboot happens. You can check the managed preferences plists like this:
defaults read /Library/Managed\ Preferences/com.apple.loginwindow
Fortunately on 10.6 and later, you can run mcxrefresh to force this propagation without a reboot:
sudo mcxrefresh -n myusername

Puppet expansion of facts inside command strings requires double quoted string

Small gotcha for puppet facts. If you want a puppet fact to be expanded inside a quoted string you need to use double quotes (i.e. a ruby-style format string).
  exec { 'something':
    # This has to be a double quote for the fact to get expanded
    command     => "/usr/bin/something ${my_custom_fact}",
    refreshonly => 'true',
  }

Wednesday, October 12, 2011

Chrome Extensions vs. Apps vs. Themes and what it looks like in your extensions directory

If you look in your chrome extensions directory on disk, things are a little confusing.

  1. There are extensions, which you can see listed at about:extensions.
  2. There are apps, which can be either packaged or hosted and will contain an app launch stanza like this in the manifest.json
       "app": {
          "launch": {
             "web_url": "https://mail.google.com/mail/mu/?mui=ca"
          },
          "urls": [ "https://mail.google.com/mail/mu/" ]
       },
    
    These don't seem to appear anywhere in the UI apart from when you open a new tab (there is no about:apps, see about:about for a full list of what is available).
  3. Themes, which are packaged in a similar way, but don't contain javascript or HTML. The manifest.json will have entries like this:
       "theme": {
          "colors": {
             "bookmark_text": [ 105, 96, 87 ],
             "frame": [ 92, 82, 73 ],
             "ntp_background": [ 235, 234, 211 ],
             "ntp_link": [ 105, 96, 87 ],
             "ntp_section": [ 158, 163, 139, 1 ],
             "ntp_section_link": [ 105, 96, 87 ],
             "ntp_section_text": [ 105, 96, 87 ],
             "ntp_text": [ 105, 96, 87 ],
             "tab_background_text": [ 255, 255, 255 ],
             "tab_text": [ 255, 255, 255 ],
             "toolbar": [ 158, 163, 139 ]
          },
    
Also, just figuring out the name of extensions can be confusing. In the manifest.json you will often see:
"name": "__MSG_some_name__",
Which means you need to go look in the relevant locale file like _locales/en/messages.json to find the actual name, it will look like this (note MSG and underscores stripped):
{
   "some_name": {
      "message": "My Rad Extension"
   }
}
The exception is if it is a theme, the messages.json file will look like:
{"themeName": {"message": "Earthy", "description": "theme name"}}
Except it also has a UTF-8 Byte Order Mark prepended to the start of the file, which is the three bytes 'ef bb bf'. If you're trying to parse it with an external JSON parser, those bytes might cause it to fail.

Friday, October 7, 2011

Using OS X directory services (dscl, dseditgroup) in single user mode

To access directory services you first need a writable root:
mount -uw /
Then you need the daemon loaded. On snow leopard:
launchctl load /System/Library/LaunchDaemons/com.apple.DirectoryServices.plist
And on Lion:
launchctl load /System/Library/LaunchDaemons/com.apple.opendirectoryd.plist

On lion you might see an error about DirectoryServices.plist not existing when you connect with dscl, it can be ignored.

Thursday, October 6, 2011

Transparent screen lock for monitoring displays

There are situations where a transparent screen lock is useful: i.e. monitoring displays, build status etc. On ubuntu I have used xtrlock, and I'm looking for something similar for OS X.

Tuesday, October 4, 2011

OS X and DNS

If you look at /etc/resolv.conf on a mac and expect to see the definitive DNS config, you are being misled. While some tools might use this config, most applications will use the mDNSResponder (the name is confusing, it is also used for unicast DNS) config maintained by configd. You can see this with:
scutil --dns
You can check out the mDNSReponder config with:
defaults read /System/Library/LaunchDaemons/com.apple.mDNSResponder
Which, incidentally, is the place to add -NoMulticastAdvertisements to disable bonjour.

You can make a query through mDNSResponder with:
$ dscacheutil -q host -a name slashdot.org
name: slashdot.org
ip_address: 216.34.181.45

$ dscacheutil -q host -a ip_address 216.34.181.45
name: slashdot.org
alias: 45.181.34.216.in-addr.arpa 
ip_address: 216.34.181.45

To see the DNS cache, dump the state of mDNSResponder into system.log with:
sudo killall -INFO mDNSResponder
You can also turn on (very verbose) DNS logging into system.log:
sudo killall -USR1 mDNSResponder
Or even turn on packet capture with:
sudo killall -USR2 mDNSResponder
You can see the other (non-DNS) caching in the local directory service with:
dscacheutil -statistics
dscacheutil -cachedump -entries

Monday, October 3, 2011

Automount and directory service search paths on OS X

The OS X directory service uses the search path to retrieve owner, group, and automount information. A config looks like this:
$ dscl localhost -read /Search
CSPSearchPath: /Local/Default /BSD/local /LDAPv3/main.ldap.example.com
DHCPLDAPDefault: off
LSPSearchPath: /Local/Default /BSD/local
NSPSearchPath: /Local/Default /BSD/local
ReadOnlyNode: ReadOnly
SearchPath: /Local/Default /BSD/local /LDAPv3/main.ldap.example.com
SearchPolicy: dsAttrTypeStandard:CSPSearchPath
The SearchPolicy tells you which one will be used - in this case the Custom Search Path or CSPSearchPath (LSPSearchPath and NSPSearchPath are read-only), which will look in /Local/Default then /BSD/local then in the LDAP server main.ldap.example.com. This is roughly the equivalent of nsswitch.conf on linux. You can also see the same information in Directory Utility:
"/System/Library/CoreServices/Directory Utility.app/Contents/MacOS/Directory Utility"
This app is used to configure the services that the mac will talk to for authentication (LDAP, NIS, Local, AD etc.), you can access that config on the command line with:
sudo defaults read /Library/Preferences/DirectoryService/DirectoryService
Automounts are kept in /etc/auto_master and you can get it to look in LDAP for the mounting info by adding '+auto_master' to the config. It will look for an automount map (see /etc/autofs.conf for more configuration options):
dn: ou=auto.master,ou=automount,ou=admin,dc=example,dc=com
ou: auto.master
objectClass: top
objectClass: automountMap
And shares identified by cn:
dn: cn=/home,ou=auto.master,ou=automount,ou=admin,dc=example,dc=com
cn: /home
objectClass: top
objectClass: automount
automountInformation: -nosuid home.nfs:/home
You can dump a full list of shares with a LDAP search like this:
ldapsearch -LLLx -b ou=auto.master,ou=automount,ou=admin,dc=example,dc=com
or via dscl like this:
dscl localhost -readall /LDAPv3/main.ldap.example.com/Automount

Friday, September 16, 2011

LDAP search queries

Some quick examples of common LDAP search queries. See this blog for more explanation and examples (thanks for the comment)

Search for a particular user:
ldapsearch -LLLx "uid=myuser"
If your LDAP database is giant, you might want to limit that search to just the people tree:
ldapsearch -LLLx -b ou=people,dc=myorg,dc=com "uid=myuser"
Find a netgroup:
ldapsearch -LLLx -b ou=netgroup,dc=myorg,dc=com "cn=mymachine.myorg.com"
Wildcards also work:
ldapsearch -LLLx -b ou=netgroup,dc=myorg,dc=com "cn=*fred*.myorg.com"
Regular group:
ldapsearch -LLLx -b ou=group,dc=myorg,dc=com "cn=sysadmins"

Reading MCX local_computer policies on the command line

Mactech has an article on using local mcx policies. To read them:
dscl . -mcxread /Computers/local_computer
and you can also see the full contents of the local_computer record like this:
defaults read /var/db/dslocal/nodes/Default/computers/local_computer

Wednesday, September 14, 2011

Reading and modifying OS X plist files on the command line

OS X uses plist files to store configuration information - you can think of them as OS X's version of the windows registry.

To read a plist you can use:
defaults read /Library/Preferences/com.apple.CrashReporter
and similarly to write a value to a plist
sudo defaults write /Library/Preferences/com.apple.CrashReporter SomeKey -bool TRUE
sudo defaults write /Library/Preferences/com.apple.CrashReporter SomeKey -string "somevalue"
and delete
sudo defaults delete /Library/Preferences/com.apple.CrashReporter SomeKey
You can also use plutil to dump an XML version to stdout:
plutil -convert xml1 -o - filename.plist
as well as convert between binary and XML formats. Xcode also ships with a plist editor (standalone prior to XCode 4, built-in since then).

And if that wasn't enough there's also PlistBuddy, here's an example command to print a particular key 'BluetoothVersionNumber':
/usr/local/bin/PlistBuddy -c Print:BluetoothVersionNumber /Library/Preferences/com.apple.Bluetooth.plist

Managing OS X users, groups, and access control from the commandline

User info
dscl . read /Users/testing
Group info
dscl . read /Groups/admin
Create a user
sudo dscl . -create /Users/testing
Add a user to the admin group
sudo dseditgroup -o edit -a testing -t user admin
Remove a user from the admin group
sudo dseditgroup -o edit -d testing -t user admin
Delete a user
sudo dscl . -delete /Users/testing

Friday, September 9, 2011

Redirection of stdout and stderr

Some notes on redirection, this article has a good summary. I hold these patterns in my head, but had long since forgotten what the syntax actually meant. Send stdout and stderr to the same file:
./generatesomeoutput &> some_file
Send stderr to stdout:
./generatesomeoutput 2>&1
which is filedescriptor '2' (stderr) to file descriptor '1' (stdout), and the '&' indicates that '1' is a filedescriptor not a filename.

Wednesday, September 7, 2011

Mounting DMG images, the hard way

Opening a DMG in the finder is a fast way to get it mounted. The manual steps are broken down as follows:

Attach the disk image (if you omit the 'nomount' flag it will do the mounting automatically, I'm stopping that for demonstration purposes)
root# hdiutil attach -nomount testing.dmg
/dev/disk1           Apple_partition_scheme          
/dev/disk1s1         Apple_partition_map             
/dev/disk1s2         Apple_Driver_ATAPI              
/dev/disk1s3         Apple_HFS                
The disk now exists:
dhcp-172-26-92-208:images root# diskutil list
[...snip...]
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     Apple_partition_scheme                        *4.2 GB     disk1
   1:        Apple_partition_map                         30.7 KB    disk1s1
   2:         Apple_Driver_ATAPI                         2.0 KB     disk1s2
   3:                  Apple_HFS Mac OS X                4.2 GB     disk1s3
And you can mount all partitions like this:
diskutil mountDisk /dev/disk1
or just one with:
diskutil mount /dev/disk1s3
Since we didn't specify targets, they will appear under /Volumes.

HOWTO Convert an Apple disk image (DMG) to ISO

hdiutil convert /path/to/filename.dmg -format UDTO -o /path/to/savefile.iso

Sunday, September 4, 2011

HOWTO: dd a disk and watch progress

On linux, dd a disk and watch progress with this command. $! is the PID of the most recent background command:
dd if=/dev/zero of=/dev/null bs=512k&
while [ $! ]; do kill -USR1 $! && sleep 10 ; done

OS X uses a different signal:
dd if=/dev/zero of=/dev/null bs=512k&
while [ $! ]; do kill -SIGINFO $! && sleep 10 ; done

Thursday, September 1, 2011

python datetime, strptime, strftime: working with datetime strings

I frequently find myself wanting to convert between a datetime object and a string representation. Here is a quick code example of swapping between the two:

t=datetime.datetime.now()
astring = t.strftime("%Y-%m-%d %H:%M:%S")
new_t = datetime.datetime.strptime(astring, "%Y-%m-%d %H:%M:%S")

Tuesday, August 23, 2011

Using FileVaultMaster.keychain key recovery on FileVault2 Lion FDE

Update Sep 2012: apple has finally released some filevault doco that describes the architecture and this process.

In OS X 10.7 Lion, Apple introduced full disk encryption in the form of FileVault 2. Apple provides a means to recover encrypted disks using 24 character recovery codes like this:


There is a more corporate version that uses asymmetric keys in a similar way to FileVault 1 (see 'Using FileVault Master Keychain'). First, create the FileVaultMaster keychain
$ sudo certtool y c k=/Library/Keychains/FileVaultMaster.keychain
...0 certificates found
...0 CRLs found
Then add a public/private key pair. The process seems to require that the labels and/or the CN of the certificate begin with the string 'FileVault Recovery Key'.
$ sudo certtool c k=/Library/Keychains/FileVaultMaster.keychain o=/Library/Keychains/FileVaultMaster.cer
Enter key and certificate label: FileVault Recovery Key (myhost.mydomain)        

Please specify parameters for the key pair you will generate.

  r  RSA
  d  DSA
  f  FEE
  e  ECDSA

Select key algorithm by letter: r

Valid key sizes for RSA are 512..2048; default is 512
Enter key size in bits or CR for default: 2048

You have selected algorithm RSA, key size 2048 bits.
OK (y/anything)? y
Enter cert/key usage (s=signing, b=signing AND encrypting, d(derive AND sign): b
...Generating key pair...

Please specify the algorithm with which your certificate will be signed.

  5  RSA with MD5
  s  RSA with SHA1

Select signature algorithm by letter: s

You have selected algorithm RSA with SHA1.
OK (y/anything)? y
...creating certificate...

You will now specify the various components of the certificate's
Relative Distinguished Name (RDN). An RDN has a number of 
components, all of which are optional, but at least one of 
which must be present. 

Note that if you are creating a certificate for use in an 
SSL/TLS server, the Common Name component of the RDN must match
exactly the host name of the server. This must not be an IP
address, but the actual domain name, e.g. www.apple.com.

Entering a CR for a given RDN component results in no value for
that component.

Common Name       (e.g, www.apple.com) : FileVault Recovery Key (myhost.mydomain)
Country           (e.g, US) : 
Organization      (e.g, Apple Computer, Inc.) : 
Organization Unit (e.g, Apple Data Security) : 
State/Province    (e.g, California) : 
Email Address     (e.g, johngalt@rand.com) : 

You have specified:
  Common Name       : FileVault Recovery Key (myhost.mydomain)
Is this OK (y/anything)? y
Wrote 765 bytes of CSR to /Library/Keychains/FileVaultMaster.cer
..cert stored in Keychain.
You can view the contents of the keychain with dump-keychain
$ sudo security dump-keychain /Library/Keychains/FileVaultMaster.keychain 
keychain: "/Library/Keychains/FileVaultMaster.keychain"
class: 0x00000010 
attributes:
    0x00000000 =0x00000010 
    0x00000001 =0x46696C655661756C74205265636F76657279204B6579202863383A32613A31343A34333A30353A33352900  "FileVault Recovery Key (myhost.mydomain)\000"
    0x00000002 =
    0x00000003 =0x00000001 
    0x00000004 =0x00000000 
    0x00000005 =0x00000000 
    0x00000006 =0xC2BF20DAFFB583D6D0B61DFB8AB67A09F73CE879  "\302\277 \332\377\265\203\326\320\266\035\373\212\266z\011\367<\350y"
    0x00000007 =
    0x00000008 =0x7B38373139316361322D306663392D313164342D383439612D3030303530326235323132327D00  "{87191ca2-0fc9-11d4-849a-000502b52122}\000"
    0x00000009 =0x0000002A  "\000\000\000*"
    0x0000000A =0x00000800 
    0x0000000B =0x00000800 
    0x0000000C =0x0000000000000000 
    0x0000000D =0x0000000000000000 
    0x0000000E =0x00000001 
    0x0000000F =0x00000001 
    0x00000010 =0x00000001 
    0x00000011 =0x00000000 
    0x00000012 =0x00000000 
    0x00000013 =0x00000001 
    0x00000014 =0x00000000 
    0x00000015 =0x00000001 
    0x00000016 =0x00000000 
    0x00000017 =0x00000000 
    0x00000018 =0x00000000 
    0x00000019 =0x00000000 
    0x0000001A =0x00000001 
keychain: "/Library/Keychains/FileVaultMaster.keychain"
class: 0x80001000 
attributes:
    "alis"="FileVault Recovery Key (myhost.mydomain)"
    "cenc"=0x00000003 
    "ctyp"=0x00000001 
    "hpky"=0xC2BF20DAFFB583D6D0B61DFB8AB67A09F73CE879  "\302\277 \332\377\265\203\326\320\266\035\373\212\266z\011\367<\350y"
    "issu"=0x30353133303106035504030C2A46696C655661756C74205265636F76657279204B6579202863383A32613A31343A34333A30353A333529  "051301\006\003U\004\003\014*FileVault Recovery Key (myhost.mydomain)"
    "labl"="FileVault Recovery Key (myhost.mydomain)"
    "skid"=
    "snbr"=0x659E5B0A  "e\236[\012"
    "subj"=0x30353133303106035504030C2A46696C655661756C74205265636F76657279204B6579202863383A32613A31343A34333A30353A333529  "051301\006\003U\004\003\014*FileVault Recovery Key (myhost.mydomain)"

Copy the keychain somewhere for escrow/backup:
$ cp /Library/Keychains/FileVaultMaster.keychain /Volumes/usbdisk/
Delete the private key out of the keychain using the 'Keychain Access' GUI. This must be done, or FileVault2 will refuse to recognise the keychain.
$ sudo /Applications/Utilities/Keychain\ Access.app/Contents/MacOS/Keychain\ Access 
You should now see only the public cert:
$ sudo security dump-keychain /Library/Keychains/FileVaultMaster.keychain 
keychain: "/Library/Keychains/FileVaultMaster.keychain"
class: 0x80001000 
attributes:
    "alis"="FileVault Recovery Key (myhost.mydomain)"
    "cenc"=0x00000003 
    "ctyp"=0x00000001 
    "hpky"=0xC2BF20DAFFB583D6D0B61DFB8AB67A09F73CE879  "\302\277 \332\377\265\203\326\320\266\035\373\212\266z\011\367<\350y"
    "issu"=0x30353133303106035504030C2A46696C655661756C74205265636F76657279204B6579202863383A32613A31343A34333A30353A333529  "051301\006\003U\004\003\014*FileVault Recovery Key (myhost.mydomain)"
    "labl"="FileVault Recovery Key (myhost.mydomain)"
    "skid"=
    "snbr"=0x659E5B0A  "e\236[\012"
    "subj"=0x30353133303106035504030C2A46696C655661756C74205265636F76657279204B6579202863383A32613A31343A34333A30353A333529  "051301\006\003U\004\003\014*FileVault Recovery Key (myhost.mydomain)"
For recovery you want to be able to mount the encrypted disk. Boot into a lion CD/USB disk/netboot image (you need lion since it has a new version of diskutil with the corestorage commands).
-bash-3.2# diskutil cs list
CoreStorage logical volume groups (1 found)
|
+-- Logical Volume Group 0483D899-2A53-4113-B8AA-6F11416AEB4C
    =========================================================
    Name:         os
    Sequence:     1
    Free Space:   0 B (0 B)
    |
    +-< Physical Volume 6E70D8B2-BBE0-42F9-BBAD-F472882FC6C2
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk0s2
    |   Status:   Online
    |   Size:     108549537792 B (108.5 GB)
    |
    +-> Logical Volume Family 7021584E-3C31-46FC-AB8C-DAFFBF660818
        ----------------------------------------------------------
        Sequence:               8
        Encryption Status:      Locked
        Encryption Type:        AES-XTS
        Encryption Context:     Present
        Conversion Status:      Complete
        Has Encrypted Extents:  Yes
        Conversion Direction:   -none-
        |
        +-> Logical Volume 18878163-252F-49B3-B4BF-D923D30CB797
            ---------------------------------------------------
            Disk:               -none-
            Status:             Locked
            Sequence:           4
            Size (Total):       108230766592 B (108.2 GB)
            Size (Converted):   -none-
            Revertible:         Yes (unlock and decryption required)
            LV Name:            os
            Content Hint:       Apple_HFS
Unlock the encrypted logical volume:
-bash-3.2# diskutil coreStorage unlockVolume 18878163-252F-49B3-B4BF-D923D30CB797 -recoveryKeyChain /Volumes/usbdisk/FileVaultMaster.keychain
Started CoreStorage operation
Logical Volume successfully unlocked
Logical Volume successfully attached as disk14
Logical Volume successfully mounted as /Volumes/os
Core Storage disk: disk14
Finished CoreStorage operation
-bash-3.2# ls /Volumes/os/
.DS_Store       .Trashes        .fseventsd      Applications    Network         Users           bin             dev             home            net             sbin            usr
.Spotlight-V100 .file           .vol            Library         System          Volumes         cores           etc             mach_kernel     private         tmp             var

Thursday, August 11, 2011

Is this Mac OS X app compiled with ASLR?

To determine if an OS X Mach-o binary is compiled with ASLR, look for the Position Independent Executable (PIE) flag in the Mach header:
$ otool -hv MobileSafari 

MobileSafari:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
   MH_MAGIC     ARM         V7  0x00     EXECUTE    40       4560   NOUNDEFS DYLDLINK TWOLEVEL PIE

Tuesday, August 9, 2011

Python: add an entry to an dict of arrays or create one if it doesn't exist

I frequently am working with dictionaries of arrays and want to add an element to an existing array, or create one if it doesn't exist. There is a neat, if not particularly easy to understand, way to do this with setdefault:

result = {}
for thisthing in alist:
   result.setdefault(thisthing.somekey, []).append(thisthing.somevalue)
So you will end up with something like:
{'key1':[value1, value2], 'key2': [value3]}

Friday, July 29, 2011

HOWTO install OS X from a system image

To install OS X from a system image, first partition the disk using diskutil:
diskutil partitionDisk disk0 2 GPT JHFS+ os 90% JHFS+ installer 10%
Download the install image onto the os partition:
curl -o /Volumes/os/osximage.dmg -L http://mydomain.com/osximage.dmg
and restore it to the installer partition:
asr restore --erase --noprompt --noverify --target /dev/disk0s3 --source /Volumes/os/osximage.dmg
Reboot!

Tuesday, July 26, 2011

64-bit errors in real programs

There is a great article on Intel's website about real programming errors in C/C++ that surfaced as bugs when the code was compiled and run as a 64-bit program. A fair proportion involve some sort of type casting that makes assumptions about pointer size, and many are subtle bugs that will only be triggered when memory allocation is > 4GB.

As 64-bit becomes the new norm I think we'll see a lot of buffer overflows and memory corruption bugs as a consequence of errors like these.

Thursday, July 21, 2011

Mac boot options

To enter single-user, hold down ⌘-S at boot.

To get a menu of boot options (netboot, USB etc.) hold down option during boot.

See the full list of boot options.

Remount a hfs partition read-write on a mac

To remount a partition read-write on linux you would use:
mount -o remount,rw /dev/disk /mnt/point
On a mac the equivalent is:
mount -u -w /dev/disk /mnt/point
In single user mode to get a writable root:
mount -uw /

Ruby learnings

I had a brief foray into Ruby land recently, so thought I would record some lessons learnt here.

Interactive ruby is like the really poor cousin to ipython. Commandline completion is not on by default, you want to enable it using:

alias irb='irb -r irb/completion'
and apparently the only way to get syntax highlighting is to install the wirble gem.

There are a bunch of ways to run shell commands. The one I used was %x:

cmd = "dscl . -read #{file} #{key} 2> /dev/null"
  output = %x{#{cmd}}

I was looking for a equivalent of the python string function 'startswith'. The ruby equivalent actually has a ? in the function name (WTF?):

if output.start_with?("#{key}: ")

Tuesday, July 12, 2011

Useful Mac keyboard shortcuts

General
Fn-arrowkeysPgUp and PgDown
Control-Shift-⏏Lock screen (sleep)
⌘-SpaceOpen Spotlight
Terminal
ESC-.Paste last word of the last command under the cursor (equivalent of Alt-. in linux)
⌘-Shift-arrowkeysLeft right arrows switch between tabs
Chrome (full list of shortcuts)
⌘-Option-arrowkeysLet right arrows to switch between tabs.

Wednesday, July 6, 2011

HOWTO enable remote desktop for windows 7 users

The technet article has all the details on how to configure remote desktop, but the tl;dr to enable it for a user is:

Control Panel -> System -> Remote Settings -> Select Users

HOWTO list local administrators on the Windows commandline

To get a list of local admins:

net localgroup administrators

Monday, June 13, 2011

Installing the shitty Amazon MP3 downloader on Ubuntu Lucid

While it sucks to have to use a MP3 downloader app at all, it's good Amazon actually has a linux version (it wasn't always the case). But it is a shame they did such a bad job of it. They wrote the downloader for Ubuntu Jaunty (now a number of releases old) and haven't updated it since.

The correct way to distribute this app would have been to provide a PPA you could put in your apt sources, which would be updated for each new version of Ubuntu, and allow the full package management capability. By using .debs any dependencies need to be manually satisfied, and you can't push updates or bug-fixes, although it looks like Amazon has written that functionality into the program itself. The deb also depends on very specific versions of libboost, rather than just specifying a minimum version number, which would allow it to work on later releases.

So installing it on lucid, or any other version is a bit of a pain. Here is one solution, that grabs the original dependencies from launchpadlibrarian and installs them:

wget https://launchpadlibrarian.net/26959932/libboost-signals1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959936/libboost-thread1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959922/libboost-iostreams1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959918/libboost-filesystem1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959916/libboost-date-time1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/26959928/libboost-regex1.34.1_1.34.1-16ubuntu1_i386.deb \
https://launchpadlibrarian.net/34165098/libicu40_4.0.1-2ubuntu2_i386.deb
sudo dpkg -i *.deb
sudo dpkg -i amazonmp3.deb

Tuesday, May 24, 2011

Contacting file owners: using find, awk, and xargs to creating lists of files and operate on them

Say you want to contact a bunch of file owners to do some cleanup or housekeeping. This will get you a list of files sorted by owner.

find . -type f -name "*blah*" "%u %M %g %y %p\n" | sort

But we can do better. Here is a neat little awk command that will drop filenames into txt files named after the user.

find . -type f -name "*blah*" "%u %M %g %y %p\n" | sort | awk 'BEGIN {FS=" "} { print $5 >>"/home/me/temp/"$1".txt" }'
So the end result is that /home/me/temp/mike.txt contains a list of files that mike owns. You can then tell mike to do something with those files (in this example, delete) using xargs like this:

xargs -a mike.txt rm -f

How do you tell mike? well you can get a list of addresses to drop into your mail client:

find . -type f -name "*blah*" "%u@company.com\n" | sort | uniq

then drag and drop the USERNAME.txt files into the mail. With a bit more effort you could automate the sending mail step too.

Friday, May 20, 2011

Adding other arguments when using *args and **kwargs in python

Sometimes you need to use *args and **kwargs to pass your positional and keyword args to another class. But what if your class takes an additional argument you don't want to pass to the super class? Here are a couple of options.

Add a new positional argument
  def __init__(self, a_list, *args, **kwargs):
    super(MyClass, self).__init__(*args, **kwargs)
    self._mylist = a_list

Add a new keyword argument
  def __init__(self, *args, **kwargs):
    self._mylist = kwargs.pop('a_list', ['some', 'default', 'list'])
    super(MyClass, self).__init__(*args, **kwargs)

Monday, May 16, 2011

HOWTO: delete/rename files with special characters (! ~ %) on the linux commandline

Sometimes you (or your code) screws up and you end up with a filename that is hard to address on the filename because it has special characters in its name. In some cases you can get away with prepending a ./ or quoting, or escaping with backslash. When those don't work, this can get you out of trouble.

First, grab the file or directory's inode:
ls -lhi
Then use the find command with the inode of the troublemaker:
find . -type f -inum 25167125 -exec mv {} binary_safe \;

Thursday, May 5, 2011

HOWTO tunnel through multiple boxes with SSH

Making ssh tunnels, even through multiple machines, is easy. These two commands will effectively link localhost:9999 with box3:80, via box2.

user@box1:~$ ssh -L 9999:localhost:9999 box2
user@box2:~$ ssh -L 9999:localhost:80 box3
If you just want to ssh through one machine to another, as of OpenSSH 5.4 there is a better way (I haven't tested this yet):
 $ cat ~/.ssh/config
 Host internalhost.myhouse.com
   ProxyCommand ssh -W %h:%p sshgateway.myhouse.com
So when you ssh to internalhost.myhouse.com you will connect through sshgateway.myhouse.com. You can chain these together, so if there was a third host only accessible from internalhost that would look like:
 $ cat ~/.ssh/config
 Host internalhost.myhouse.com
   ProxyCommand ssh -W %h:%p sshgateway.myhouse.com

 Host ilovelayers.myhouse.com
   ProxyCommand ssh -W %h:%p internalhost.myhouse.com

Wednesday, May 4, 2011

Loving the Cyber Bomb? Why don't we love the Cyber Security Bounties instead?

A paper Loving the Cyber Bomb? The Dangers of Threat Inflation in Cybersecurity Policy by Brito and Watkins of the Mercatus Institute at Virginia's George Mason University makes the argument that the cyber threat is being overblown by government agencies and defence contractors in the chase for dollars. It draws some analogies with the 'evidence' of WMDs that led to the invasion of Iraq.

I don't really buy the analogy that the cyber threat is as vaporous as Iraq's WMDs. Brito and Watkins seem to be implying that the argument that anything less than the ability of an attacker to "derail trains, release chlorine gas, or bring down the power grid" doesn't matter that much. We have plenty of evidence that companies of all sizes are regularly compromised, and I'm sure they'd tell you the loss of business, and/or costs incurred, mattered. I think the Estonian government and businesses would also argue with Brito and Watkins' downplaying of the 2007 DDOS. A cited lack of previous power outages provably due to electronic attack to-date doesn't mean it isn't possible, and it might be more likely to happen at the consumer level as the industry moves towards smart metering.

However, Brito and Watkins do highlight the dangers of poorly targeted government spending being poured into defence contractor's pockets for little gain in security. It was interesting to read about the courtship of the newly-established US Cyber Command ('Cyber Pork' p. 26) by various US towns and states, in a bid to attract its billions of dollars of government investment. Maryland eventually won that battle.

How could these dollars be spent to get the best security for your dollar? I would suggest bounties for specific, measurable, and product-agnostic security improvements. Something like the items from this list produced by the Australian government. Government could offer to cover the costs of implementation (up to a fixed amount) of the top 5 security controls. The offer could be restricted to select government agencies, as well as companies running important infrastructure such as power and water. If you spent $500k on an application whitelisting rollout for a power company it would seem cheap when the next Conficker rolled around.

Monday, May 2, 2011

HOWTO mount a remote luks encrypted volume on demand

I wanted to create a cron'd backup to a luks volume on a remote machine. My preference was to not have the volume mounted automatically so if my friend rebooted the box it wouldn't block the boot process waiting for the password. It would also be nice if it just mounted itself when necessary and was locked for the rest of the time.

First authorise my user to mount and unlock the volume using specific sudo commands (in /etc/sudoers):

Cmnd_Alias CRYPTOPEN=/sbin/cryptsetup luksOpen /dev/disk/by-uuid/41885992-3f80-4aaa-bc60-9c5854017ca9 crypt-backup --key-file /tmp/keyfile
Cmnd_Alias MOUNT=/bin/mount /dev/mapper/crypt-backup /mnt/backup
Cmnd_Alias UMOUNT=/bin/umount /mnt/backup
Cmnd_Alias CRYPTCLOSE=/sbin/cryptsetup luksClose crypt-backup

myuser ALL=(root) NOPASSWD: CRYPTOPEN,MOUNT,UMOUNT,CRYPTCLOSE

Then, a script on my side:

#!/bin/sh

scp /data/backup/scripts/backup/hdd_keyfile.luks home:/tmp/keyfile && \
ssh home "chmod 600 /tmp/keyfile && sudo /sbin/cryptsetup luksOpen /dev/disk/by-uuid/41885992-3f80-4aaa-bc60-9c5854017ca9 crypt-backup --key-file /tmp/keyfile"
if [ $? -ne 0 ]; then
    echo "cryptsetup failed."
    ssh home "shred -u /tmp/keyfile"
    exit 1
fi

ssh home "sudo /bin/mount /dev/mapper/crypt-backup /mnt/backup"
if [ $? -ne 0 ]; then
    echo "mount failed."
    exit 1
fi

rsync -rtv --compress-level=4 /data/ home:/mnt/backup/data/
rsync -rtv --compress-level=4 /mp3/ home:/mnt/backup/mp3/

ssh home "sudo /bin/umount /mnt/backup && sudo /sbin/cryptsetup luksClose crypt-backup"
if [ $? -ne 0 ]; then
    echo "umount failed."
    exit 1
fi

Chuck it in a cron. Done.

Monday, April 25, 2011

Dropbox authentication issues

There has been some discussion of whether the dropbox authentication system is a vulnerability, or just as insecure as everything else. At the very least: if changing your password doesn't change the underlying key stored on your machine, that an attacker could have copied, it makes denying access to an attacker post-compromise fairly difficult.

Wednesday, April 13, 2011

#ifdef DEBUG print statements in C

The #ifdef DEBUG design pattern for debug output is extremely common in C code. Here is a nice DEBUG macro I have used in the past:

#ifdef DEBUG
#define DEBUG_PRINT(fmt, args...)  \
       fprintf(stderr, "%s:%d:%s(): " fmt, __FILE__, __LINE__, __FUNCTION__, ##args)
#else
#define DEBUG_PRINT(fmt, args...)    /* Don't do anything in non-debug builds */
#endif

As suggested in this post, letting the compiler see your debug code is important to ensure it isn't broken by people changing code without making debug builds. This macro achieves that goal by letting the preprocessor insert a statement that will be pulled out by the optimiser for non-debug builds. This is the best solution I have come across.

#ifdef DEBUG
#define DEBUG_TEST 1
#else
#define DEBUG_TEST 0
#endif

#define DEBUG_PRINT(fmt, args...) \
        do { if (DEBUG_TEST) fprintf(stderr, "%s:%d:%s(): " fmt, __FILE__, \
                                __LINE__, __FUNCTION__, ##args); } while (0)

Mingw example makefile for console app and DLL

Here is a basic Makefile to use with mingw to compile some common types of windows binaries - a windows console app, a non-console 'windows' subsystem app, and a DLL.

# Example mingw makefile

TARGETNAME=myapp

CC = i586-mingw32msvc-gcc

# A basic windows app (won't bring up a console window)
CFLAGS = -Wall -O -Wl,--subsystem,windows

# for debugging make this a console app (so we can write to stdout/stderr) easily;
# turn off optimisations and include symbols to help with running in a debugger
CFLAGS_DEBUG = -Wall -O0 -g -Wl,--subsystem,console -DDEBUG

# Create a dll instead of an exe
CFLAGS_DLL = -Wall -O0 -g -shared -Wl,--subsystem,windows -DDLL

# Any libraries you need - in this case wininet for communicating to the internet
LIBS = -lwininet

# Strip the binary for our prod build
STRIP = i586-mingw32msvc-strip

# UPX pack to minimise size for our prod build
UPX=upx -9

# Leave symbols, turn off optimisation, and send output to console
debug: $(TARGETNAME).c
        $(CC) -o $(TARGETNAME)_dbg $(CFLAGS_DEBUG) $^ $(LIBS)

# No debug symbols, debug statements removed, subsystem windows means we won't pop a window
$(TARGETNAME): $(TARGETNAME).c
        $(CC) -o $@ $(CFLAGS) $^ $(LIBS)

$(TARGETNAME).dll: $(TARGETNAME).c
        $(CC) -o $@ $(CFLAGS_DLL) $^ $(LIBS)

release: $(TARGETNAME)
        $(STRIP) $<
        $(UPX) $<

Using a word macro to run an executable on Windows 7, Office 2007

To write a macro in Office 2007, you'll want the developer tab in the ribbon.  Turn it on by clicking the Office icon in the top left corner of Word and then 'Word Options'.  Find the 'Show Developer tab in the Ribbon' and check it.

Use this macro code to run blah.exe when the document is opened (using AutoOpen):

Private Declare Function WinExec Lib "Kernel32.dll" (ByVal lpCmdLine As String, ByVal nShowCmd As Long) As Long

Sub AutoOpen()

    Call WinExec("C:\blah.exe", SW_NORMAL)

End Sub

Sunday, April 3, 2011

Streaming internet radio with MythTV

Streaming internet radio with mythtv is, sadly, still confined to the realm of hacks. One simple solution is to just drop ".pls" files into your videos directory. Assuming you use mplayer to play your videos, that should just work. Unfortunately I like to use the internal player so I can use the time stretch feature on videos as well as recordings. The internal player doesn't play ".pls" files, so that option is out.

The other approach is to hack in a new menu item and get it to run mplayer. There are a few descriptions of how to do this, so here is my 2c.

Edit /usr/share/mythtv/themes/defaultmenu/library.xml to add:

<button>
    <type>MUSIC_RADIO</type>
    <text>Listen to Internet Radio</text>
    <description>Listen to Internet Radio</description>
    <action>MENU inetradio.xml</action>
    <depends>mythmusic</depends>
</button>

And create inetradio.xml in the same directory.  The Australian Broadcasting Corporation (ABC) links are a little hard to find - you can get them here.
<mythmenu name="INETRADIO">

<button>
<type>MUSIC_PLAY</type>
<text>NPR</text>
<action>EXEC /usr/bin/mythradio http://npr.ic.llnwd.net/stream/npr_live24</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>TripleJ</text>
<action>EXEC /usr/bin/mythradio http://www.abc.net.au/res/streaming/audio/aac/triplej.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>RawFM</text>
<action>EXEC /usr/bin/mythradio http://www.rawfm.com.au/stream/playlists/160kbitMP3.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>666 ABC Canberra</text>
<action>EXEC /usr/bin/mythradio http://www.abc.net.au/res/streaming/audio/aac/local_canberra.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>ABC Classic FM</text>
<action>EXEC /usr/bin/mythradio http://www.abc.net.au/res/streaming/audio/aac/classic_fm.pls</action>
</button>

<button>
<type>MUSIC_PLAY</type>
<text>Stop Playing</text>
<action>EXEC pkill -f mplayer</action>
</button>

</mythmenu>

And create /usr/bin/mythradio (this wouldn't be necessary if you could give multiple commands to "EXEC" - unfortunately it seems to break the syntax and mythtv says it is an invalid XML file). The include file is a one line mplayer config that tells mplayer to allow the screensaver to run, so you can avoid burn-in while listening to the radio.

#!/bin/bash
pkill -f mplayer
/usr/bin/mplayer -include /etc/mythtv/mplayer_noscr.conf $1 &

and here is /etc/mythtv/mplayer_noscr.conf:

stop-xscreensaver=no

Saturday, March 12, 2011

HOWTO Troubleshoot windows sound problems (for parents using google video chat)

The good people at google, have a nice parent-friendly intro to video chat. Unfortunately, getting to the point of having a gmail account with the plugin installed, is, in my experience, the easy part.

The bigger challenge is troubleshooting audio problems, and the google troubleshooting page doesn't offer much help, apart from a weird endorsement of Logitech products.  For the people I chat with, the problem is usually a muted microphone, and/or muted sound.  Instructions for modifying sound setup on Windows 7 are below.

Look at the bottom right corner of the screen for the speaker icon


Left click the speaker, and make sure the volume is turned up


Check if that solves the problem.  If not, right click the speaker and choose playback devices


Click Properties


and check the slider is turned up on the Levels tab.



 
Click OK, and get out to the desktop.
 
Right click on the speaker again to check your microphone.  Click recording devices.
 
 
 The green bar should move up and down as it registers sound.  Try talking.

 
If that all looks OK, google provides an interface for you to check your settings from within gmail.  
 
Login and click on the settings link on the top right corner


Then click on the chat tab


and on the + sign next to "Verify your settings".


If you answer 'Yes' to all three questions in that box, and you still don't have working audio or video, you can tell the person on the other side that it is a problem on their end, and point them to these instructions :)

HOWTO Create multiple chrome and firefox profiles on linux

I like using multiple browser profiles for security reasons. I keep my valuable cookies, such as those for email etc. in one profile, and do regular web-browsing in another. This means if I hit a XSS through regular browsing, the scope for cookie stealing is limited. When using firefox with the noscript plugin it also makes sense to have different whitelists for these two usage scenarios.

Creating firefox profiles is simple, just use the profile manager, which you can bring up with:
firefox --no-remote -P

Once you have created the profile, I copy the shortcut icon and change the target to point at the new profile like this:
firefox %u --no-remote -P browse

Chrome is almost as easy, but the doco can be a little harder to find. You just need to specify a new user data directory like this:

/usr/lib/chromium-browser/chromium-browser --enable-udd-profiles --user-data-dir=.config/google-chrome/Browse

and change your shortcut icon to run the same command.

On Ubuntu natty and oneiric this got harder with the new launcher. There are a lot of ways to get it done, but this seemed the simplest to me:
sudo apt-get install alacarte gnome-panel
  • Start "Main Menu" by searching for it in the dash.
  • Add an item for your application and close the editor.
  • Logout to refresh the dash (there must be a way to do this without a logout, but not sure what it is).
  • Start the application by searching for it in the dash.
  • Right-click on the icon and "Keep in launcher"

I also created a couple of (ugly) custom icons to keep my 'Browse' and 'Mail' profiles clearly marked.

Monday, January 24, 2011

HOWTO add a directory to your python path

You can add to your python path like this:

import sys
sys.path.append("/home/me/mypy") 

The (python) way of the future: pip and virtualenv

I recently discovered the magic of pip and virtualenv. Pip is shaping up as a serious contender for replacement of distutils and easy_install. I haven't used it much yet, and will have to reserve judgement until I package a decent-sized project with pip.

Using pip is easy, although there is a bootstrapping problem that, ironically, I used easy_install to solve. If your distro packages a recent version of pip, you can use it to skip this bit.

sudo apt-get install python-setuptools
s easy_install pip
s pip install virtualenv

This gives you a virtualenv install. Virtualenv is brilliant, and has significantly influenced the way I write python. Virtualenv allows you to build a complete python environment in a directory that you 'activate' by running an in-built shell script that sets your python path environment variables. The are many advantages to this approach, but here are a few I have encountered:
  • Simplified testing - test your code against multiple versions of libraries and utilities
  • Keep separate dev and production environments on your dev machine, allowing you to test in a production environment without pushing the code to your actual production server.
  • The isolation provided by Virtualenv ensures you aren't depending on libraries on your dev box that aren't installed in production, and upgrading libraries won't break other applications

You can even create virtual environments with different versions of the python interpreter using:
virtualenv -p python2.6 2.6ENV

So, once you have a virtualenv, activate it and install yolk to give you a picture of what is inside the environment.
source .testenv/bin/activate
~> pip install yolk
~> yolk -l
Python          - 2.6.4        - active development (/usr/lib/python2.6/lib-dynload)
distribute      - 0.6.14       - active
pip             - 0.8.1        - active
wsgiref         - 0.1.2        - active development (/usr/lib/python2.6)
yolk            - 0.4.1        - active

Then go ahead and install the other packages you need:

pip install django

You can also build a requirements file to install exactly the same versions somewhere else:
pip freeze -E .testenv > requirements.txt

The holy trinity is apparently pip, virtualenv and fabric. Fabric is something I'll have to play with soon. There is a good tutorial on pip and virtualenv here, and using them with django here.

The only limitation/problem I have run into is that you can't move the virtual environment once it is created. There is a "--relocatable" directive to address this, but the webpage says that it is somewhat experimental at the moment.

Syntax highlighting code in Blogger

Posting code in Blogger by default results in something that is inexcusably ugly.  How can you make code posted in blogger look pretty?  For me, syntax highlighting and scrollable text boxes are a must.

The solution is: google-code-prettify, which does the highlighting for code.google.com (you could also consider SyntaxHighlighter).  Now the next problem is, how do you distribute the javascript and CSS for prettify?  Unfortunately Google doesn't make it available via their CDN like they do for other libraries like jQuery.  This would obviously be the best option for users since:
  • it would negate the need for additional hosting; and 
  • minimise load time since browsers could cache the code across many sites, and if needed download it from the closest CDN node.
So the remaining options are to slap it into your blogger template, host it yourself somewhere (like Google pages), use the subversion browse code link (ick), or figure out what code.google.com does, since it uses the same code. 

I opted for the last option, which works a treat.  Put this in your blogger template, right after the <head> tag:

<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/ph_core.css">
 
<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/ph_detail.css" >
 
<link type="text/css" rel="stylesheet" href="http://www.gstatic.com/codesite/ph/3799605220899551948/css/d_sb_20080522.css" >

Then add an onload event to the body tag to call the PrettyPrint function:

<body onload="prettyPrint()" >

and, since the prettyprint CSS doesn't give you a scrolling box by default (!?), add this to the CSS in your template to avoid your code being chopped off:

/* Override prettyprint CSS, which doesn't allow scrolling box */
pre.prettyprint {
  overflow: auto;
}

Once you have done all that, you just need to wrap the code in your posts in:

<pre class="prettyprint">
</pre>

In summary, much harder than it needs to be. Blogger team, please build this functionality in!

Wednesday, January 19, 2011

List of DNS Providers

Just came across a list of DNS providers I made a while ago, and thought I'd post it here. They all seem to still be in business.  The top three I have heard good reports about:
  1. DNS Made Easy
  2. changeip.com 
  3. zoneedit.com
  4. bur.st (non-profit based in Perth)
  5. dnspark.com
  6. xname.org

    GPG HOWTO

    Here is a good, quick intro to using GPG, including generating and publishing keys, encrypting and signing.

    Tuesday, January 18, 2011

    The Google Apps administrator control panel

    The Google Apps administrator control panel for "your_domain.com" can be found at the fairly un-intuitive address:

    https://www.google.com/a/your_domain.com
    

    Wednesday, January 12, 2011

    Linux boot process digram

    Finally got around to converting my RHCE notes on the linux boot process into a diagram.

    Wednesday, January 5, 2011

    Creating a HTML DIV that adjusts its size to automatically fit the content

    To create a div that automatically adjusts its size to fit the content, use this in your CSS:
    margin: 0 auto;
    
    Using 'margin' for this purpose doesn't seem particularly intuitive to me, and isn't mentioned in CSS doco I read, but it works.

    Rendering a django queryset as a html table (like form.as_table) using a custom template filter

    Django has some nice helpers for forms (forms.as_p and forms.as_table) that you can use in your template to minimise boilerplate HTML. Unfortunately, despite some discussion/confusion, and at least one project, there doesn't appear to be a similar elegant solution for querysets.

    My approach to the problem was to implement a custom template filter you can use in a template like this (where object_list is your queryset):
    {{ object_list|result_as_table }}
    
    i.e. you hand it a queryset, and it gives you back a table.

    Here is the code for the filter. I'm using this for a number of subclasses of a class called resultdata, so that they can share an identical template and just provide different querysets in urls.py.

    def result_as_table(queryset, fieldnames=None):
        """Take a resultdata queryset and return it as a HTML table.
        Columns will be returned in the order they are declared in the model.
        
        Optionally, provide a list of fieldnames, which will be used to limit the output."""
    
    
        dictlist = queryset.values()
        output = "<table>\n"
        output_keys = []
    
        if fieldnames:
            names = fieldnames
        else:
            names = dictlist.field_names
    
        for name in names:
    
            if not name.endswith("_id"):
                output_keys.append(name)
                output = "".join( (output, "<th>%s</th>\n" % escape(name.replace("_"," ").capitalize())) )
    
        for rddict in dictlist:
    
            output = "".join( (output, "<tr>\n") )
    
            for key in output_keys:
                val = rddict[key]
                if not key.endswith("_id"):
    
                    display_func = get_display_method(rddict, queryset, key)
    
                    if display_func:
                        output = "".join( (output, "<td>%s</td>\n" % escape(display_func())) )
                    else:
                        output = "".join( (output, "<td>%s</td>\n" % escape(val)) )
    
            output = "".join( (output, "</tr>\n") )
    
        return mark_safe("".join( (output, "</table>\n") ))
    
    result_as_table.is_safe = True
    

    This filter introspects the column headers, strips out any id fields (since users don't care about them), converts underscores to spaces in column headers, and finds the corresponding display methods for any 'choices' fields so you can still get the nice human-readable output. It outputs HTML, so it uses django's own 'django.utils.html.escape' method and marks the output as safe so django doesn't escape all the HTML.

    Finding the display methods was a little harder than I expected, since the python inspect code was dying when given a django model object. This code is unfortunately tied to the resultdata object due to the naming of the keys in the values() dictionary, making it less generic than I would like.

    def get_display_method(rddict, queryset, key):
        """Re-implementation of inspect.getmembers(rddict,inspect.ismethod) and 
        a test to see if this is a get_field_status method.  Had to reimplement inspect
        because it was bombing on the object - expecting some property that is not present."""
    
        rdobj = queryset[0].__class__.objects.filter(resultdata_id = rddict["resultdata_id"]).get()
        targetname = "get_%s_display" % key
        display_func = False
    
        for name in dir(rdobj):
            if name == targetname:
                try:
                    display_func = getattr(rdobj, name)
                    break
                except Exception,e:
                    pass
    
        return display_func
    

    There was a major gotcha with using the field_names property of the queryset.values() dictionary. If you passed a queryset with annotations like this is your urls.py:

    list_rd = {
        'extra_context': {'fieldnames': ['status','first','total']},
        'queryset': ResultData.objects.annotate(total = Count('link__clickback'), 
                                                first = Min('link__clickback__timestamp')),
    }
    
    urlpatterns += patterns('project.app.views',
        url(r'^result/detail/$', 'detailed_results', kwargs=list_rd, name='detailed_results'),
    )
    
    

    The annotations would not get added into the field_names property. This was solved by providing the optional fieldnames parameter.