Tim's blag

To content | To menu | To search

Monday 4 March 2013

Clone an OS X volume using rsync

There are several commercial tools for cloning hard disks for OS X, including SuperDuper! and Carcon Copy Cloner. However, the same can be achieved by using rsync.

I wrote a script based on necolas' rsync_backup. It has several improvements over necolas' version, but the core is the same. I used Carbon Copy Cloner's list of files to exclude when cloning.

I tested the script on my machine using an external USB hard disk on OS X 10.7 and after cloning for a few hours (like the first Time Machine backup), I can simply boot off the USB disk by holding the 'Option' key. rsync can make incremental updates, such that next clones are much faster (~10 minutes in my case).

N.B. I do not guarantee anything for this script, if it breaks your computer: tough luck. There are some safe guards in the script, but I refuse to promise anything. I recommend to read through the script and verify that it works before use.

Sunday 28 October 2012

ZSH/Bash string manipulation

Both ZSH and Bash have some built-in string manipulations that can be useful at times.


Given a test string:

word='hello - world - how - are - you'

Take elements (letters) 5 through 10

$word[5,10] == 'o - wo' # for zsh
${word:5:5} == '- wo'   # for bash

N.B.: in bash the first character is at index 0, in zsh this is at index 1!

Substring matching

To match a string with a substring, we can use:

  • ${str#substr} Deletes shortest match of $substr from front $str
  • ${str##substr} Deletes longest match of $substr from front $str

for example: take part before 'how*', or o*

${word%how*} == 'hello - world - '
${word%o*} == 'hello - world - how - are - y'
${word%%o*} == 'hell'

Or take the part after '* - '

${word#* - } = 'world - how - are - you'
${word##* - } = 'you'

Explode strings

To explode or split strings into array, we can use several zsh functions:

To split by ' - ', take 3rd element

${word[(ws: - :)3]} == 'how' # only zsh

Split by ' - ', take second element, capitalize

${(C)${word[(ws: - :)2]}} == 'World' # only zsh

Split by word, take first word

$word[(w)1] == 'hello' # only zsh

File rename examples

Get the file extension:

${file##*\.} == jpg
echo ${file##*\.}

Get the file base name:

${file%\.*} == helloworld.multiple.dots
echo ${file%\.*}


Sunday 14 October 2012

Utrecht University Solis VPN on Mac OS X Lion 10.7.*

Utrecht University offers a VPN service called Solis-VPN. This works fine under Windows using the Cisco Client, but this client is not available for newer (10.6+) version of Mac OS X that run a 64-bit kernel by default.

Update: Cisco released the 'Cisco Anyconnect' VPN client available that works under OS X 10.7.* (Lion) as well. It is a Java client which Solis-VPN users can download at https://solisvpn.uu.nl. Unfortunately it seems, this client only routes traffic over the VPN that is on the target network, there is no option to route all traffic over the VPN. Therefore, you cannot route your web traffic over the VPN to download papers from journals that only allow access from a university network.

Although OS X Lion has a built-in VPN client that should support the Cisco protocol, this did not work in my case with the UU Solis VPN for some reason. I finally got it to work using vpnc. The following steps should help you set up the VPN client as well:

  1. Download and install Xcode and the Developer Tools. The latter can be installed at the Downloads section of Xcode's preferences.
  2. Install Macports, a package manager for OS X
  3. Install vpnc
  4. Configure tuntaposx
  5. Configure vpnc
  6. Connect and disconnect the VPN

This worked for me on Mac OS X 10.7.5 with vpnc version 0.5.3 and tuntaposx version 20111101.

Step 1 and 2 are documented in abundant details elsewhere, so I will skip these here.

Install vpnc (step 3)

Install vpnc with the hybrid_cert option in case you might need it. For UU Solis VPN this is not required.

sudo port install vpnc +hybrid_cert

Configure tuntaposx (step 4)

To set up the network connection, vpnc uses tuntaposx. This is automatically installed once you install vpnc, but it's not configured properly for some reason. To fix this, run

sudo rsync -av /opt/local/Library/Extensions/* /Library/Extensions/
sudo rsync -av /opt/local/Library/StartupItems/* /Library/StartupItems/

which will copy the Kernel extension bundles to the proper locations and load them at startup.

Configure vpnc (step 5)

To connect to your favourite VPN server, create a configuration file in /opt/local/etc/vpnc/. For Solis VPN, you need these settings:

IPSec gateway
IPSec ID everyone
IPSec secret rlakluxiupoeqlurlaqo
IKE Authmode psk
Xauth username <SOLISID>
Xauth password <PASSWD>
Enable Single DES
DPD idle timeout (our side) 0
Local Port 0

Note that you can write your password here such that vpnc will connect automatically, but I left it empty (just delete the Xauth password line) such that vpnc asks for my password at startup.

Also, IPSec secret is a group password that Cisco VPN uses to authenticate the user to some group, or something. This is the decoded version, obtained with the Cisco vpnclient password decoder. For Solis VPN, the encoded password is:


Connect and disconnect (step 6)

To run vpnc, simply type

sudo vpnc --natt-mode cisco-udp solis-vpn

and vpnc will connect to the VPN. To disconnect, run

sudo vpnc-disconnect

and you are again on your own.


Monday 20 August 2012

Apple Time Capsule benchmarks

I got a 4th generation Apple Time Capsule a while ago and here are some benchmarks that I took.

Continue reading...

OpenProj export as PNG/SVG

OpenProj is one of the few project management software programs available on the Mac platform, intending to replace Microsoft Project. Unfortunately it is impossible to easily export schemes as PDF for use in publications. OpenProj was forked to ProjectLibre which promises to release a new and improved version in August 2012.

Until that time, there are a few people who wrote patches for OpenProj so you can fix it yourself. openprojext adds an 'export to PNG' feature for OpenProj, and Eduardo Suarez-Santana extended this patch to also allow 'export as SVG', which subsequently allows conversion to PDF. The latter patch worked fine for me so I put up a patched version of OpenProj 1.4 online. It is patched exactly according to the instruction on Eduardo's website, using Batik version 1.7 and XML Commons External version 1.3.05. To start it, run

 java -jar openproj.jar

There is also an even more extended patch for OpenProj called Angelfalls, but I did not get this running easily and export to SVG was all I needed anyway.

Tuesday 20 December 2011

Advanced ImageMagick use: image arithmetic and colorizing

ImageMagick is a suite of tools used to manipulate images of various formats (including FITS files!). I use ImageMagick's convert utility to convert images from one format to another. For example

$ convert rose.jpg rose.png


$ convert rose.jpg -resize 50% -quality 50 rose_thumb.jpg

Image arithmetic

But besides simple image conversions, ImageMagick can also perform arithmetic on images, for example average, subtract or divide images (which can be used for dark- or flat-fielding). In the case you have several flatfield images you want to average, simply use the -average flag

$ convert flat001.png flat00n.png flat003.png -average flat_avg.png

Alternatively, you can specify the output format to be different from the input as well.

To use this flatfield to process images (divide raw images by the flatfield), use the -compose and -composite flags.

$ convert raw001.png flat_avg.png -compose Divide_Src \
  -composite -auto-level raw001-flat.png

Here we use raw001.png as input and divide it by flat_avg.png (-compose Divide_Src -composite). After that we equalize the image such that all values fit within the file format (-auto-level) and store it to disk (raw001-flat.png).

Instead of using -auto-level, you could also use the -evaluate flag to scale the image in a more predictable way, for example if you want to convert the images to a movie later on. In this case the previous line would read something like

$ convert raw001.png flat_avg.png -compose Divide_Src \
  -composite -evaluate multiply 0.5 raw001-flat.png

to scale the composite image by a factor 0.5. -evaluate can also be used to add, subtract or do other things.

Colorize images

ImageMagick can also be used to colorize images with an arbitrary gradient (in fact it can perform any function on any image with the -fx flag).

First we generate a discrete gradient (which will automatically be interpolated later):

$ convert -size 1x1 xc:#FFFFE5 xc:#FFF7BC xc:#FEE391 xc:#FEC44F xc:#FB9A29 \
  xc:#EC7014 xc:#CC4C02 xc:#993404 xc:#662506 \
  +append -rotate -90 paul_tol_linear_clut.png

(we used Paul Tol's optimal linear color scheme here). In a nutshell, this command makes several images (xc:) of 1 by 1 pixel (-size 1x1) of one color. The results are appended and then rotated -90 degrees before being stored to disk as paul_tol_linear_clut.png. It is important that the color gradient is vertical.

Once we have a gradient, we can use it to colorize an image:

$ convert test_image.png paul_tol_linear_clut.png \
  -fx 'v.p{0,u*v.h}'  test_image-colorized.png

Unforunately the -fx flag is rather slow because it is very general in nature. The -clut option should be faster and easier, but I did not get it to work.

$ convert -clut -interpolate bicubic test_image.png  paul_tol_linear_clut.png \

Paul Tol's color schemes

For reference, I include Paul Tol's optimal color schemes here as well, both as discrete look up table and as gradient. The LUTs can be used to colorize any image as described above. The gradients are for illustration only.

Linear color map

convert -size 1x1 xc:#FFFFE5 xc:#FFF7BC xc:#FEE391 xc:#FEC44F xc:#FB9A29 \
  xc:#EC7014 xc:#CC4C02 xc:#993404 xc:#662506 +append \
  -rotate -90 paul_tol_linear_clut.png
convert -size 1x1 xc:#FFFFE5 xc:#FFF7BC xc:#FEE391 xc:#FEC44F xc:#FB9A29 \
  xc:#EC7014 xc:#CC4C02 xc:#993404 xc:#662506 +append \
  -filter Cubic -resize 600x50! paul_tol_linear_grad.png


Plus-minus color map

convert -size 1x1 xc:#3D52A1 xc:#3A89C9 xc:#77B7E5 xc:#B4DDF7 xc:#E6F5FE \
  xc:#FFFAD2 xc:#FFE3AA xc:#F9BD7E xc:#ED875E xc:#D24D3E xc:#AE1C3E \
  +append -rotate -90 paul_tol_plusmin_clut.png
convert -size 1x1 xc:#3D52A1 xc:#3A89C9 xc:#77B7E5 xc:#B4DDF7 xc:#E6F5FE \
  xc:#FFFAD2 xc:#FFE3AA xc:#F9BD7E xc:#ED875E xc:#D24D3E xc:#AE1C3E \
  +append -filter Cubic -resize 600x50! paul_tol_plusmin_grad.png


Rainbow color map

convert -size 1x1 xc:#781C81 xc:#42368F xc:#3F5CAB xc:#4580C1 xc:#519CB8 \
  xc:#5DA8A4 xc:#6DB388 xc:#94BD5F xc:#C5BA45 xc:#DEA63A xc:#E78833 \
  xc:#E55F2B xc:#D92120 +append -rotate -90 paul_tol_rainbow_clut.png
convert -size 1x1 xc:#781C81 xc:#42368F xc:#3F5CAB xc:#4580C1 xc:#519CB8 \
  xc:#5DA8A4 xc:#6DB388 xc:#94BD5F xc:#C5BA45 xc:#DEA63A xc:#E78833 \
  xc:#E55F2B xc:#D92120 +append -filter Cubic -resize 600x50! paul_tol_rainbow_grad.png



More information on:

English Spelling

I found a rather brilliant British English poem and added the pronunciation next to it in IPA. I used Apple's Dictionary Service available in Python to parse all words and extract the pronunciation.

English Spelling screenshot

Thursday 17 November 2011

Python meets C: Cython

This bit is about optimizing code Python code using something that's closer to the metal (i.e. a CPU/GPU). Before you do anything about optimization, realize this:

Early Optimization is the root of all evil - Donald Knuth

If you still think you need to optimize your code, read on.

Continue reading...

OSX launchd and launchctl issues

I recently re-installed Lion on my MBP and ran into the problem that by default ssh-agent remembers my private key indefinitely. I wanted to reduce this to something sane and found the file /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist which controls ssh-agent's behaviour. I started tweaking stuff with launchctl, first broke my ssh-agent altogether and after a while figured out how to tweak it to my needs.

After initially breaking my ssh-agent because I copy/pasted commands that I didn't really understand, I found the following apt quote:

A good rule for rocket experimenters to follow is this: always assume that it will explodeAstronautics, issue 38, October 1937 (source)

Fortunately I managed to fix it and get some basic understandign of launcd on OSX. This information seemed to be a bit fragmented, so here are my findings.

Continue reading...

Monday 7 November 2011

Analyse written text

Earlier I wrote a simple script that checks a text for structural integrity, making sure there are no words close together etc. I converted this script to Python, dubbed it metacheck.py and I present the new and improved version here!

Continue reading...

Python introduction 101

I recently gave a introductory presentation to Python to some colleagues. The topics covered include the basic language syntax of Python, available data types, some words about memory management and some sightly more advanced Python features like list comprehension and array slicing. Click further to read more details.

Continue reading...

Sunday 6 November 2011

iPhone 3GS battery usage

I had some trouble with my iPhone 3GS where the standby time and usage time was the same, so I started monitoring the battery usage. I solved the intial problem (fully resetting the whole phone helps) but thought it'd be interesting to continue monitoring battery usage and analyse the results. The results are presented below.

Continue reading...

Tuesday 28 June 2011

Change Airport & Time Machine with Applescript - update

Earlier I wrote a bit about changing mouse preferences on your Mac. I added some other settings which might change from location to location using Applescript, such as Airport and Time Machine backup disk.

Continue reading...

Monday 27 June 2011

Custom math fonts in XeTeX

I am a rather avid user of the LaTeX markup language, but I have recently begun switching to XeTeX, mostly for it's superior font handling and support. Although it is fairly trivial to use arbitrary (OpenType) fonts in XeTeX, when I tried to use arbitrary fonts for math mode this proved to be a little trickier.

Continue reading...

PyAna Python library

PyAna is a Python library for reading and writing (Rice-compressed (pdf)) ANA files. I wrote this when I was working with ANA files a lot, but this is no longer the case. Therefore, I am not maintaining the code any longer. The current version seems to be quite stable (no known memory-leaks), albeit a bit rough on the edges.

The library wraps some ancient C routines into a NumPy module. See PyAna @ github for more details. I wrote this with help from this NumPy recipe and the NumPy book.

Besides a useful library, this can also be used as boilerplate code if you want to write your own Numpy module.

Unix output, pipes

Sometimes you might want to diff the output of two commands instead of two files. To do this, simply call

diff <(ls /tmp/folder1) <(ls /tmp/folder2)

See this Linuxjournal article and this blogpost for more details on pipes etc.

Sunday 26 June 2011

Optimal colour schemes

As Paul Tol writes:

Graphics with scientific data become clearer when the colours are chosen carefully. It is convenient to have a good default scheme ready for each type of data, with colours that are distinct for all readers, including colour-blind people.

and to that goal he has redesigned several different colour schemes from the ground-up to maximize visibility for both colour-blind and colour-aware people. The result is a set of colour schemes for various applications which make best use of colour space. There is no reason not to use them, so take a look at Paul's technote.

Continue reading...

Analyse word distance in text

Update: I wrote an improved version in Python, read more here.

When I was writing my thesis a few years ago, I wrote a script to find the word distance for each word, and signal words that are close together. I used this to find obvious flaws in texts without having to read it over every time. Especially if you are revising a text a lot, this might be useful to see if you need to use a thesaurus somewhere.

Continue reading...

Thursday 23 June 2011

Machiner precision: float vs double

I found a small code snippet on machine precision for float and double datatypes in C. Might be interesting for some people. I adapted the code slightly and put it online in a gist. I reproduced the code here for clarity.

Other interesting posts on this topic include this Stackoverflow topic which in turn refers this appendix on floating-point arithmetic..

Continue reading...

Tuesday 21 June 2011

Old version Skype for Mac

Because Skype for Mac version 5.x sucks monkey balls, here's how to downgrade to version 2.8:
There is also a tutorial available if you need step-wise instructions. If you don't know why you're reading this check here and here.

- page 1 of 2