Getting notification when a long-running command finishes.

My job often involves running a command to do something, realising that it's going to take longer than I expected and going doing something else in the meantime. Conversely I'll run something that should take a long time, go do something else and find that it stopped with an error after a very short time and I didn't notice.

For a long time I've wanted to set things up so that any command that takes more than 20s to run would send me an IM when it was finished. What I didn't want was to have to prefix all my commands with some notification because that's annoying and I would probably forget in exactly the cases I need it.

It sounds pretty simple but getting it work turned out to be quite fiddly.

Delivering an instant message.

There's a program call sendxmpp that will deliver an instant message over XMPP. It's written in Perl and definitely works on Linux but probably also Mac and Windows if you really want. Every few months I would try to make it work and when I finally got it working, it would stop working again the next day.. Apparently Google's XMPP service is particularly finicky. I appear to have made it work for real now. I honestly don't know what I did differently, it may even be something that changed when Google deprecated XMPP in favour of Hangouts. As I write this, I see that it no longer works for one of my accounts but it's working fine with another! If there's a more reliable option, I'd love to know.

To make this work you'll need a second Google account because sending a message from yourself to yourself doesn't seem to work. So I created an account that I'm going to use only for sending IMs. After creating that account, I invited it to chat and made sure that the accounts can chat with each other through the normal Google chat interface.

Next I created a config file for it with the following commands (you'll need to replace some parts of the last one)

touch ~/.sendxmpp chmod 600 ~/.sendxmpp echo my.new.im.username@gmail.com;talk.google.com MyNewPassword > ~/.sendxmpp

Now to send a notification to myself I just do

echo some message | sendxmpp -t -u my.new.im.username -o gmail.com other.username@gmail.com

and the message shows up. I also get a bunch of Perl warnings but such is life.

Finally I wrapped that up as a command called notify and I can just pipe a message into that.

Spotting that a command is taking a long time to finish

I hate unix shells, they're a horrible mishmash of special cases, obscure features and crap that's only that way for backwards compatibility. What I've done is a hack in bash and it's not perfect (it seems like it would be slightly less hacky in zsh but that wouldn't fix the oddities described below). The core is this

trap bash_pre_cmd DEBUG PROMPT_COMMAND=bash_post_command

What that does is run bash_pre_cmd every time you hit enter to kick off a command and bash_post_command every time it prints the command prompt (which indicates that the command has finished, although there are various way for this not to be true, hence the oddities but in normal usage it's true). So all that remains is sensible definitions for those two commands

# Prefix things with "fergal_" to avoid name clashes. # Sets $fergal_timer to the current value of $SECONDS if it doesn't already have # a value. For whatever reason, this get called multiple times per command typed # in the shell, so we only want to start once. function fergal_start_timer () { if [[ -z $fergal_timer ]]; then fergal_timer=$SECONDS fi return 0 } # Calls fergal_timer_callback with two arguments: # 1 the number of seconds elapsed since fergal_start_timer was called # 2 the last command. # Safely handles the case where $fergal_timer was not yet set (e.g. on # startup). function fergal_end_timer () { if [[ -n $fergal_timer ]]; then fergal_timer_callback $(($SECONDS - $fergal_timer)) \ "$(fc -ln -1 | sed -e 's/^[ \t]*//')" fergal_timer= fi } function bash_pre_cmd () { fergal_start_timer } function bash_post_command () { record_eternal_history fergal_end_timer } # Notify if it takes more than 20s but always notify for "make" unless it takes 0 seconds. function fergal_timer_callback () { local time=$1 shift local command="$1" shift local notify if echo $command | grep -q "^make\b" || [[ $time -gt 20 ]]; then notify=1 fi # Never notify if it took 0 seconds. Mostly important to avoid # notifying when the user hits enter on a blank line. if [[ $time == 0 ]]; then notify= fi if [[ $notify ]]; then # sendxmpp blurts out crap on stderr even when everything works # fine, so we suppress that. (echo Completed after ${time}s; echo $command) | \ tee >(notify > ~/.notify_ouput 2>&1 || echo >&2 notify failed) fi }

You'll probably want to customize the callback for your own preferences.

Put all that into your .bashrc and cross your fingers

Oddities

If you suspend a process, that will cause the prompt to be printed and may cause a notification. If you unsuspend that process with fg then when it finishes, this code will see fg as the command that just completed. Also, we cannot notify for 0 seconds otherwise just hitting enter would cause notification and the IM would refer to the last command in the history. Interactive commands obviously take a long time and so cause useless notifications. I'll probably whitelist a few common ones like man and less. There are probably some problems too but for normal type-run-complete shell use it works just fine.

Basically shells are shitty and not very flexible. Doing this correctly would involve being able to stash away some state along with each command that's started then getting access to that state when the command finishes, it would require changes to shell.

"fixing" a hard disk with hdrecover

Mostly so I don't forget. I fixed Midori's windows laptop a while back. After finding 0 viruses and others nasties, I couldn't explain why it was so jumpy and crappy. During the virus scan I'd noticed some disk errors and after fruitless messing with SMART I ran into hdrecovers which reads every sector of the disk and tries to provoke the disk into declaring a bad sector as irreparable. The disk then marks it as such and replaces it with one from its stash of spare sectors.

Surprisingly, after "fixing" about 10 sectors, video stopping stopping and audio stopped being randomly distorted. For whatever reason, these sectors were upsetting windows but they clearly weren't actually important because when they are fixed, their data is lost forever.

I'm about to give it a spin over another disk that had been causing problems.

Tae Kim's guide to Japanese Grammar as a .epub

Here it is.

This guide to Japanese grammaris highly recommended by lots of people because it teaches "real" Japanese rather than a stilted textbook version and also because it doesn't try to over-translate the Japanese example sentences, rather it just leaves them making their sense in Japanese. This last bit is particularly interesting as the "true" meaning of a sentence in Japanese is often highly dependent on the context because in Japanese you're allowed leave out pretty much anything that is obvious from the context, subject, object etc. so picking a single meaning as the translation is often just not appropriate.

I've been meaning to read it for ages but I don't want to print it out and most of my reading now is on my Android phone, e.g. while putting Seán to sleep. The guide comes as a giant HTML file or a PDF, neither of which are convenient on Android. There is however a very nice ebook reader called Aldiko which can open .epub files. Despite the fact that .epub is just zipped up html files, Alidko cannot read plain HTML.

After much searching, I finally found a html to epub converter called Calibre - Google search didn't help, in the end I found it through Aldiko's help pages and it seems to have done a fine job of converting the book to .epub format with it's html2epub program (no setting options or anything, just point it at the html and it works). Calibre is open source and available to apt-get on recent Ubuntu releases. I'm pretty sure I'll be using it often. It would also make a very nice web-service - give it a URL and get back a .epub great for those too-long-to-read-right-now web pages, maybe I'll get around to something like that.

making dvds from video files

Lest I forget again, DeVeDe did a decent job of putting an flv file that I had downloaded onto a DVD for playback on any TV and it's apt-gettable on ubuntu.

Watching Japanese TV in Dublin

ssh terebi mkfifo pipe ssh terebi cat pipe | mplayer -

then in another window

ssh terebi mencoder -oac lavc -ovc lavc -lavcopts vcodec=mpeg4:vbitrate=100:acodec=ac3:abitrate=64 /dev/video0 -o pipe

gives me watchable live tv.

I think I should really be using some variety of vlc but that's another day's work

Useful things when you've wiped you partition table (conclusion)

This is the concluding part of my earlier post on recovering after wiping the partition table and the first 30k or so of my harddisk. It has a happy ending.

It didn't take much more effort in the end to rebuild my laptop's partition table. It turns out that gpart was not terribly useful as I had used extended partitions and nuked the first one. It found my 2 linux partitions but refused to rebuild the partition table. It also outputs the location of the partitions in MB which is no use at all. This patch

diff -ur gpart-0.1h/src/gpart.h ../gpart-0.1h/src/gpart.h --- gpart-0.1h/src/gpart.h 2001-01-30 23:07:29.000000000 +0000 +++ ../gpart-0.1h/src/gpart.h 2007-01-03 05:07:06.000000000 +0000 @@ -171,7 +171,7 @@ struct disk_geom *disk_geometry(disk_desc *); int reread_partition_table(int); -#define s2mb(d,s) { (s)*=(d)->d_ssize; (s)/=1024; (s)/=1024; } +#define s2mb(d,s) { (s)*=(d)->d_ssize; } #define align(b,s) (byte_t *)(((size_t)(b)+(s)-1)&~((s)-1)) #include "gmodules.h"

makes it output bytes instead which I could then divide by 512 to get sectors. I had to compile it on a Debian box (not sure what vintage), it wouldn't compile under Ubuntu.

Next I used sfdisk -d /dev/sda to dump a partition table from another machine as an example and then edit that to look like

# partition table of /dev/sda unit: sectors /dev/sda1 : start= 63, size= 39053952, Id=83 /dev/sda2 : start= 39070143, size= 3903728, Id=82 /dev/sda3 : start= 42973938, size= 72260370, Id=8e /dev/sda4 : start=115234308, size= 1975932, Id=83, bootable

I worked out the sizes by subtracting the start values from each other. Saving that in a file and doing sfdisk < my_partitions.txt gave me a new partition table which amazingly worked. Of course /dev/sda1 wasn't recovered properly because it had also been partially overwritten by the format but luckily I had nothing useful on that partition.

Making a secure bootable cd with sshd and my ssh keys

Update: ddclient doesn't seem to work. I don't have time to figure out why, the rest works fine though. Update: slax is currently "frozen", I've updated the links in this doc to point to the legacy hosting service. Also, it seems that they have messed with the sshd config and possibly broken it in the latest version.

I've built a headless box that's going to live in Japan when I go back to Ireland. There'll be no one in Japan to fix it if it goes bad. The best I can do is get someone to insert a CD and reboot but after some searching and some asking it seems none of the bootable CDs will boot the machine into a useful state for remote admin. slax was recommended to me on TLUG as a good customisable distro. It turned out to be pretty easy thanks to slax's rootcopy feature which allows you to make minor modifications without digging around (too much) in the compressed package files. I've broken the process down into a few steps, with a quick bash script for each one.

Create a directory to work in and cd into it

mkdir slax cd slax

Get a copy of the .iso

You need slax-frodo-x.y.z.iso, you can get it from the slax download page. The frodo edition is the minimal version on which all the others are based.

Make a copy of the files from the CD

This is necessary because we need to add some files. So I ran this as ./prep_image.sh ~/slax-frodo-5.1.8.iso

#! /bin/bash iso=$1 shift if [ "$iso" = '' ]; then echo 1>&2 Usage: echo 1>&2 "$0 " exit 1 fi mkdir slax.mnt mount -t iso9660 $iso slax.mnt -o loop cp -v -Tr --preserve=all slax.mnt slax umount slax.mnt

Enable SSH

It's disabled by default (rc.sshd is not executable), so I replace it with one that is executable. I also turn off password logins because everyone knows the root password for slax. Finally I include an authorized_keys file so that I can ssh in with my ssh key. I ran this script as ./enable_ssh.sh ~/.ssh/authorized_keys

#! /bin/bash auth_keys=$1 shift if [ "$auth_keys" = '' ]; then echo 1>&2 Usage: echo 1>&2 "$0 <path/to/auth_keys_file<" exit 1 fi # make rc.sshd executable and stop root logins using a password # # extract rc.sshd from the module and tweak the permissions mkdir 02_core.mnt mount -t squashfs slax/base/02_core.mo 02_core.mnt -o loop mkdir -p rootcopy/etc/rc.d cp 02_core.mnt/etc/rc.d/rc.sshd rootcopy/etc/rc.d chmod 700 rootcopy/etc/rc.d/rc.sshd # disable password logins mkdir -p rootcopy/etc/ssh cp 02_core.mnt/etc/ssh/sshd_config rootcopy/etc/ssh cat >> rootcopy/etc/ssh/sshd_config <<EOM # since we're starting sshd by default, don't allow root logins with # passwd, must use ssh keys PasswordAuthentication no EOM # install keys file mkdir -p rootcopy/root/.ssh chmod 700 rootcopy/root/.ssh cp $auth_keys rootcopy/root/.ssh chmod 600 rootcopy/root/.ssh/authorized_keys cp -r --preserve=all rootcopy/* slax/rootcopy umount 02_core.mnt

Update a dyndns entry

The machine is on a cable modem and doesn't have a fixed IP, so I need to know where it is. I downloaded the ddclient module and the following script puts it in place, puts your ddclient.conf file into /etc and makes sure that it gets started on boot (I assumed it would start by itself actually, I'm not sure if that's a bug or not). I invoked this script as ./ddclient.sh ddclient.conf

#! /bin/bash conf=$1 shift if [ "$conf" = '' ]; then echo 1>&2 Usage: echo 1>&2 "$0 " exit 1 fi cp ddclient_3_7_0.mo slax/base mkdir -p rootcopy/etc/ cp $conf rootcopy/etc/ddclient.conf mkdir -p rootcopy/etc/rc.d cat > rootcopy/etc/rc.d/rc.local <<EOM #! /bin/bash /etc/rc.d/rc.ddclient start EOM chmod 755 rootcopy/etc/rc.d/rc.local

Add any other modules

There doesn't appear to be an editor in the slax-frodo image so I grabbed joe and copied the .mo into slax/base/

Move files into place and create a new .iso

So far everything we've done has been in a rootcopy/ directory in the current directory. Now it's time to move that into place in slax/ and invoke the iso building script with the following script which outputs it to slax-ssh.iso in the current directory.

#! /bin/bash rm -rf slax/rootcopy cp -r --preserve=all rootcopy slax cd slax ./make_iso.sh ../slax-ssh.iso

Test it in qemu

qemu -cdrom slax-ssh.iso should boot up. I never figured out how to get qemu bring up a real IP interface - the virtual machine can connect to the outside world but I can't connect to it. I tested it by setting up an ssh tunnel connecting port 22 on the VM to port 2022 on the host machine with ssh -R2022:localhost:22 fergal@my.host.machine.ip. Then I did ssh -p 2022 root@localhost with my ssh keys loaded and in I went. When my keys aren't loaded, I don't get a password prompt so I can't get in. Perfect.