Tim's blag

To content | To menu | To search

Sunday 25 November 2018

Reading out TP-Link HS110 on Linux/Raspberry Pi

I bought a TP-Link HS110 version 3.0 smart wi-fi plug for 30 EUR to read out power usage of some appliances, which I want to connect to a Raspberry Pi 3B+.

Device setup

Here we connect the HS110 to our existing WiFi network without using the TP-link Kasa app and without it ever connecting to the TP-link cloud server.

  1. Plug into wall socket, the wifi symbol will blink green-amber.
  2. Get python clients https://github.com/softScheck/tplink-smartplug
  3. Connect to HS110 access point TP-LINK_Smart Plug_XXXX.
    1. Get initial information: ./tplink_smartplug.py -t -c info
  4. Disable cloud access:
    1. Set devs.tplinkcloud.com to resolve to (or similar)
    2. Set cloud server to new value: using ./tplink_smartplug.py -t -j '{"cnCloud":{"set_server_url":{"server":"xs4all.nl"}}}'. N.B. err_code:0 means no error
  5. Bind to existing WiFi network: ./tplink_smartplug.py -t -j '{"netif":{"set_stainfo":{"ssid":"WiFi","password":"123","key_type":3}}}'. N.B. This command is not encrypted so it will leak your SSID and network. There is no way around it as the official Kasa app uses the same protocol.

Device read out

Now the device is connected to our WiFi network and we can use it as a measurement device

  1. Find out new IP
    1. On your router, look at connected devices, write down IP
    2. Ping broadcast address to find IP of all connected devices responding to ping: e.g. ping, the TP-link HS110 will respond
  2. Read out device: ./tplink_smartplug.py -t -c energy


Accuracy is comparable to Brennenstuhl PM 231E offline power meter.

15W incandescent bulb

  • Reference measurement: 15.0±0.2W (average-by-eye)
  • TP-Link measurement: 15.03±0.09W (average of 10 measurements)

1300W heating iron

  • Reference measurement: 1340±5W (average-by-eye)
  • TP-link measurement: 1342W±5W (average of 14 measurements)

N.B. the heating iron's load drops 20W over the timeframe of a minute, such that these uncertainties are upper limits.

Power usage

The TP-link HS110 itself uses 4.3±0.2 Watt (as measured by Brennenstuhl PM 231E, which has a reported accuracy of 1% or 0.2W).


The plug starts up in ±8 seconds, and then takes ±3 seconds to join the WiFi network.


  1. Elgato Eve Energy Works with Bluetooth (vs WiFi), compatible with Homekit. Could work on Linux using Homebridge, but this sounded like more trouble than reading out over Wi-Fi/HTTPs
  2. Fibaro Wall Plug Works with Z-wave (vs WiFi), requires additional Z-wave dongle, which I don't have. It's also more expensive than the TP-link. After that it could work on Linux, e.g. via this OpenHab post or this Youtube Tutorial


Sunday 24 September 2017

Using PAR to protect against bit rot

Since OSX does not offer native protection against bit rot (for example with ZSH or other mature file systems), I've started using PAR2 to manually protect my photos. To do this, I've written some scripts that I'm documenting here for myself and others.

The process seems to work, however it's rather laborious. Some optimization might be useful/necessary.

Create PAR files for one directory

# Directory to create PAR for

# Directory containing metadata (e.g. log files) and par2 executable

# Go to directory, e.g. '20051203_vacation_trip'
# Extract year from directory name
# Base PAR file on year of the directory

# Remove DS_Store files, these are not important and change often
find . -type f | grep .DS_Store$ | xargs rm

# Init log file with current date
date >> ${parlogf}
# nice: reduce prio for this command 
# caffeinate: prevent disk or computer sleep when running
# par2: create parity files recursively, 1% redundancy, with 10k blocks, for all files
# tee: store output to log file and show on screen
nice -n 10 caffeinate -ms ${MAINTDIR}/par2 c -R -r1 -b10000 ${parfile} ./* | tee -a ${parlogf}

Create PAR files for many directories

Since my pictures are stored in many different directories, and since par2 is not parallelized, the above can be extended to par many directories in parallel as follows.

First we define a function which does the above, in bash:

# We need bash since it supports functions

# Define a function which pars one directory (basically script above)
function create_par {


  # Remove DS_Store files, these are not important and change often, polluting
  # the PAR2 archive
  find /Users/tim/Pictures/${CHECKDIR} -type f | grep .DS_Store$ | xargs rm

  # Move existing PAR files to archive dir
  mv *par2 ${PHOTODIR}/maintenance/0-oldpar/

  echo "Now creating PAR in $(pwd)\n========================="
  nice -n 10 caffeinate -ms ${parcmd} c -R -r1 -b10000 ${parfile} ./*

  cd ${CURDIR}

Now we can simply run

create_par 2016* | tee -a ${parlogf}

to create par files for any directory. Using the simple command parallel, we can parallelize this:

export -f create_par
parallel -j 4 -k create_par ::: {2015,2010,2009,2008,2007,2006,2005,2003}* | tee -a ${parlogf}

Check PAR files for one directory

After creating PAR files, we need to verify the integrity once in a while, which can be done as follows

function check_par {

  echo "Now checking $(pwd)\n========================="
  nice -n 5 caffeinate -ms ${parcmd} v ${parfile} | grep -v "Target.* - found.$|^Load|^Scanning: "
  cd ${CURDIR}

then call the command

check_par $PICDIR

Or run in parallel again

export -f check_par
parallel -j 4 -k check_par ::: {0,1,2}* | tee -a ${parlogf}

Monday 4 March 2013

Clone an OS X volume using rsync

There are several commercial tools for cloning hard disks for OS X, including SuperDuper! and Carcon Copy Cloner. However, the same can be achieved by using rsync.

I wrote a script based on necolas' rsync_backup. It has several improvements over necolas' version, but the core is the same. I used Carbon Copy Cloner's list of files to exclude when cloning.

I tested the script on my machine using an external USB hard disk on OS X 10.7 and after cloning for a few hours (like the first Time Machine backup), I can simply boot off the USB disk by holding the 'Option' key. rsync can make incremental updates, such that next clones are much faster (~10 minutes in my case).

N.B. I do not guarantee anything for this script, if it breaks your computer: tough luck. There are some safe guards in the script, but I refuse to promise anything. I recommend to read through the script and verify that it works before use.

Sunday 28 October 2012

ZSH/Bash string manipulation

Both ZSH and Bash have some built-in string manipulations that can be useful at times.


Given a test string:

word='hello - world - how - are - you'

Take elements (letters) 5 through 10

$word[5,10] == 'o - wo' # for zsh
${word:5:5} == '- wo'   # for bash

N.B.: in bash the first character is at index 0, in zsh this is at index 1!

Substring matching

To match a string with a substring, we can use:

  • ${str#substr} Deletes shortest match of $substr from front $str
  • ${str##substr} Deletes longest match of $substr from front $str

for example: take part before 'how*', or o*

${word%how*} == 'hello - world - '
${word%o*} == 'hello - world - how - are - y'
${word%%o*} == 'hell'

Or take the part after '* - '

${word#* - } = 'world - how - are - you'
${word##* - } = 'you'

Explode strings

To explode or split strings into array, we can use several zsh functions:

To split by ' - ', take 3rd element

${word[(ws: - :)3]} == 'how' # only zsh

Split by ' - ', take second element, capitalize

${(C)${word[(ws: - :)2]}} == 'World' # only zsh

Split by word, take first word

$word[(w)1] == 'hello' # only zsh

File rename examples

Get the file extension:

${file##*\.} == jpg
echo ${file##*\.}

Get the file base name:

${file%\.*} == helloworld.multiple.dots
echo ${file%\.*}


Sunday 14 October 2012

Utrecht University Solis VPN on Mac OS X Lion 10.7.*

Utrecht University offers a VPN service called Solis-VPN. This works fine under Windows using the Cisco Client, but this client is not available for newer (10.6+) version of Mac OS X that run a 64-bit kernel by default.

Update: Cisco released the 'Cisco Anyconnect' VPN client available that works under OS X 10.7.* (Lion) as well. It is a Java client which Solis-VPN users can download at https://solisvpn.uu.nl. Unfortunately it seems, this client only routes traffic over the VPN that is on the target network, there is no option to route all traffic over the VPN. Therefore, you cannot route your web traffic over the VPN to download papers from journals that only allow access from a university network.

Although OS X Lion has a built-in VPN client that should support the Cisco protocol, this did not work in my case with the UU Solis VPN for some reason. I finally got it to work using vpnc. The following steps should help you set up the VPN client as well:

  1. Download and install Xcode and the Developer Tools. The latter can be installed at the Downloads section of Xcode's preferences.
  2. Install Macports, a package manager for OS X
  3. Install vpnc
  4. Configure tuntaposx
  5. Configure vpnc
  6. Connect and disconnect the VPN

This worked for me on Mac OS X 10.7.5 with vpnc version 0.5.3 and tuntaposx version 20111101.

Step 1 and 2 are documented in abundant details elsewhere, so I will skip these here.

Install vpnc (step 3)

Install vpnc with the hybrid_cert option in case you might need it. For UU Solis VPN this is not required.

sudo port install vpnc +hybrid_cert

Configure tuntaposx (step 4)

To set up the network connection, vpnc uses tuntaposx. This is automatically installed once you install vpnc, but it's not configured properly for some reason. To fix this, run

sudo rsync -av /opt/local/Library/Extensions/* /Library/Extensions/
sudo rsync -av /opt/local/Library/StartupItems/* /Library/StartupItems/

which will copy the Kernel extension bundles to the proper locations and load them at startup.

Configure vpnc (step 5)

To connect to your favourite VPN server, create a configuration file in /opt/local/etc/vpnc/. For Solis VPN, you need these settings:

IPSec gateway
IPSec ID everyone
IPSec secret rlakluxiupoeqlurlaqo
IKE Authmode psk
Xauth username <SOLISID>
Xauth password <PASSWD>
Enable Single DES
DPD idle timeout (our side) 0
Local Port 0

Note that you can write your password here such that vpnc will connect automatically, but I left it empty (just delete the Xauth password line) such that vpnc asks for my password at startup.

Also, IPSec secret is a group password that Cisco VPN uses to authenticate the user to some group, or something. This is the decoded version, obtained with the Cisco vpnclient password decoder. For Solis VPN, the encoded password is:


Connect and disconnect (step 6)

To run vpnc, simply type

sudo vpnc --natt-mode cisco-udp solis-vpn

and vpnc will connect to the VPN. To disconnect, run

sudo vpnc-disconnect

and you are again on your own.


Monday 20 August 2012

Apple Time Capsule benchmarks

I got a 4th generation Apple Time Capsule a while ago and here are some benchmarks that I took.

Continue reading...

OpenProj export as PNG/SVG

OpenProj is one of the few project management software programs available on the Mac platform, intending to replace Microsoft Project. Unfortunately it is impossible to easily export schemes as PDF for use in publications. OpenProj was forked to ProjectLibre which promises to release a new and improved version in August 2012.

Until that time, there are a few people who wrote patches for OpenProj so you can fix it yourself. openprojext adds an 'export to PNG' feature for OpenProj, and Eduardo Suarez-Santana extended this patch to also allow 'export as SVG', which subsequently allows conversion to PDF. The latter patch worked fine for me so I put up a patched version of OpenProj 1.4 online. It is patched exactly according to the instruction on Eduardo's website, using Batik version 1.7 and XML Commons External version 1.3.05. To start it, run

 java -jar openproj.jar

There is also an even more extended patch for OpenProj called Angelfalls, but I did not get this running easily and export to SVG was all I needed anyway.

Tuesday 20 December 2011

Advanced ImageMagick use: image arithmetic and colorizing

ImageMagick is a suite of tools used to manipulate images of various formats (including FITS files!). I use ImageMagick's convert utility to convert images from one format to another. For example

$ convert rose.jpg rose.png


$ convert rose.jpg -resize 50% -quality 50 rose_thumb.jpg

Image arithmetic

But besides simple image conversions, ImageMagick can also perform arithmetic on images, for example average, subtract or divide images (which can be used for dark- or flat-fielding). In the case you have several flatfield images you want to average, simply use the -average flag

$ convert flat001.png flat00n.png flat003.png -average flat_avg.png

Alternatively, you can specify the output format to be different from the input as well.

To use this flatfield to process images (divide raw images by the flatfield), use the -compose and -composite flags.

$ convert raw001.png flat_avg.png -compose Divide_Src \
  -composite -auto-level raw001-flat.png

Here we use raw001.png as input and divide it by flat_avg.png (-compose Divide_Src -composite). After that we equalize the image such that all values fit within the file format (-auto-level) and store it to disk (raw001-flat.png).

Instead of using -auto-level, you could also use the -evaluate flag to scale the image in a more predictable way, for example if you want to convert the images to a movie later on. In this case the previous line would read something like

$ convert raw001.png flat_avg.png -compose Divide_Src \
  -composite -evaluate multiply 0.5 raw001-flat.png

to scale the composite image by a factor 0.5. -evaluate can also be used to add, subtract or do other things.

Colorize images

ImageMagick can also be used to colorize images with an arbitrary gradient (in fact it can perform any function on any image with the -fx flag).

First we generate a discrete gradient (which will automatically be interpolated later):

$ convert -size 1x1 xc:#FFFFE5 xc:#FFF7BC xc:#FEE391 xc:#FEC44F xc:#FB9A29 \
  xc:#EC7014 xc:#CC4C02 xc:#993404 xc:#662506 \
  +append -rotate -90 paul_tol_linear_clut.png

(we used Paul Tol's optimal linear color scheme here). In a nutshell, this command makes several images (xc:) of 1 by 1 pixel (-size 1x1) of one color. The results are appended and then rotated -90 degrees before being stored to disk as paul_tol_linear_clut.png. It is important that the color gradient is vertical.

Once we have a gradient, we can use it to colorize an image:

$ convert test_image.png paul_tol_linear_clut.png \
  -fx 'v.p{0,u*v.h}'  test_image-colorized.png

Unforunately the -fx flag is rather slow because it is very general in nature. The -clut option should be faster and easier, but I did not get it to work.

$ convert -clut -interpolate bicubic test_image.png  paul_tol_linear_clut.png \

Paul Tol's color schemes

For reference, I include Paul Tol's optimal color schemes here as well, both as discrete look up table and as gradient. The LUTs can be used to colorize any image as described above. The gradients are for illustration only.

Linear color map

convert -size 1x1 xc:#FFFFE5 xc:#FFF7BC xc:#FEE391 xc:#FEC44F xc:#FB9A29 \
  xc:#EC7014 xc:#CC4C02 xc:#993404 xc:#662506 +append \
  -rotate -90 paul_tol_linear_clut.png
convert -size 1x1 xc:#FFFFE5 xc:#FFF7BC xc:#FEE391 xc:#FEC44F xc:#FB9A29 \
  xc:#EC7014 xc:#CC4C02 xc:#993404 xc:#662506 +append \
  -filter Cubic -resize 600x50! paul_tol_linear_grad.png


Plus-minus color map

convert -size 1x1 xc:#3D52A1 xc:#3A89C9 xc:#77B7E5 xc:#B4DDF7 xc:#E6F5FE \
  xc:#FFFAD2 xc:#FFE3AA xc:#F9BD7E xc:#ED875E xc:#D24D3E xc:#AE1C3E \
  +append -rotate -90 paul_tol_plusmin_clut.png
convert -size 1x1 xc:#3D52A1 xc:#3A89C9 xc:#77B7E5 xc:#B4DDF7 xc:#E6F5FE \
  xc:#FFFAD2 xc:#FFE3AA xc:#F9BD7E xc:#ED875E xc:#D24D3E xc:#AE1C3E \
  +append -filter Cubic -resize 600x50! paul_tol_plusmin_grad.png


Rainbow color map

convert -size 1x1 xc:#781C81 xc:#42368F xc:#3F5CAB xc:#4580C1 xc:#519CB8 \
  xc:#5DA8A4 xc:#6DB388 xc:#94BD5F xc:#C5BA45 xc:#DEA63A xc:#E78833 \
  xc:#E55F2B xc:#D92120 +append -rotate -90 paul_tol_rainbow_clut.png
convert -size 1x1 xc:#781C81 xc:#42368F xc:#3F5CAB xc:#4580C1 xc:#519CB8 \
  xc:#5DA8A4 xc:#6DB388 xc:#94BD5F xc:#C5BA45 xc:#DEA63A xc:#E78833 \
  xc:#E55F2B xc:#D92120 +append -filter Cubic -resize 600x50! paul_tol_rainbow_grad.png



More information on:

English Spelling

I found a rather brilliant British English poem and added the pronunciation next to it in IPA. I used Apple's Dictionary Service available in Python to parse all words and extract the pronunciation.

English Spelling screenshot

Thursday 17 November 2011

Python meets C: Cython

This bit is about optimizing code Python code using something that's closer to the metal (i.e. a CPU/GPU). Before you do anything about optimization, realize this:

Early Optimization is the root of all evil - Donald Knuth

If you still think you need to optimize your code, read on.

Continue reading...

OSX launchd and launchctl issues

I recently re-installed Lion on my MBP and ran into the problem that by default ssh-agent remembers my private key indefinitely. I wanted to reduce this to something sane and found the file /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist which controls ssh-agent's behaviour. I started tweaking stuff with launchctl, first broke my ssh-agent altogether and after a while figured out how to tweak it to my needs.

After initially breaking my ssh-agent because I copy/pasted commands that I didn't really understand, I found the following apt quote:

A good rule for rocket experimenters to follow is this: always assume that it will explodeAstronautics, issue 38, October 1937 (source)

Fortunately I managed to fix it and get some basic understandign of launcd on OSX. This information seemed to be a bit fragmented, so here are my findings.

Continue reading...

Monday 7 November 2011

Analyse written text

Earlier I wrote a simple script that checks a text for structural integrity, making sure there are no words close together etc. I converted this script to Python, dubbed it metacheck.py and I present the new and improved version here!

Continue reading...

Python introduction 101

I recently gave a introductory presentation to Python to some colleagues. The topics covered include the basic language syntax of Python, available data types, some words about memory management and some sightly more advanced Python features like list comprehension and array slicing. Click further to read more details.

Continue reading...

Sunday 6 November 2011

iPhone 3GS battery usage

I had some trouble with my iPhone 3GS where the standby time and usage time was the same, so I started monitoring the battery usage. I solved the intial problem (fully resetting the whole phone helps) but thought it'd be interesting to continue monitoring battery usage and analyse the results. The results are presented below.

Continue reading...

Tuesday 28 June 2011

Change Airport & Time Machine with Applescript - update

Earlier I wrote a bit about changing mouse preferences on your Mac. I added some other settings which might change from location to location using Applescript, such as Airport and Time Machine backup disk.

Continue reading...

Monday 27 June 2011

Custom math fonts in XeTeX

I am a rather avid user of the LaTeX markup language, but I have recently begun switching to XeTeX, mostly for it's superior font handling and support. Although it is fairly trivial to use arbitrary (OpenType) fonts in XeTeX, when I tried to use arbitrary fonts for math mode this proved to be a little trickier.

Continue reading...

PyAna Python library

PyAna is a Python library for reading and writing (Rice-compressed (pdf)) ANA files. I wrote this when I was working with ANA files a lot, but this is no longer the case. Therefore, I am not maintaining the code any longer. The current version seems to be quite stable (no known memory-leaks), albeit a bit rough on the edges.

The library wraps some ancient C routines into a NumPy module. See PyAna @ github for more details. I wrote this with help from this NumPy recipe and the NumPy book.

Besides a useful library, this can also be used as boilerplate code if you want to write your own Numpy module.

Unix output, pipes

Sometimes you might want to diff the output of two commands instead of two files. To do this, simply call

diff <(ls /tmp/folder1) <(ls /tmp/folder2)

See this Linuxjournal article and this blogpost for more details on pipes etc.

Sunday 26 June 2011

Optimal colour schemes

As Paul Tol writes:

Graphics with scientific data become clearer when the colours are chosen carefully. It is convenient to have a good default scheme ready for each type of data, with colours that are distinct for all readers, including colour-blind people.

and to that goal he has redesigned several different colour schemes from the ground-up to maximize visibility for both colour-blind and colour-aware people. The result is a set of colour schemes for various applications which make best use of colour space. There is no reason not to use them, so take a look at Paul's technote.

Continue reading...

Analyse word distance in text

Update: I wrote an improved version in Python, read more here.

When I was writing my thesis a few years ago, I wrote a script to find the word distance for each word, and signal words that are close together. I used this to find obvious flaws in texts without having to read it over every time. Especially if you are revising a text a lot, this might be useful to see if you need to use a thesaurus somewhere.

Continue reading...

- page 1 of 2