Robust Bash Scripts - Part Four

This is part of a series of posts which should aid anyone writing Bash scripts to make them more robust.

This post describes how you can use traps to help your code fail more gracefully.

Sometimes you write a script, it fails, and leaves a shed load of temporary files over your file system.  Or perhaps (like in part two) some stuff gets set up, your script fails and leaves a network half set up or a naming service half completed.  Whatever the problem, we can trap them (well, almost all of them…).

For my example, I’m going to use a section of my dissertation.  I have a script which is called “INSTALL.sh”, this sets some stuff up itself, and also delegates some tasks to other scripts.  Here is the general gist:

Check for root permissions - If no permissions, exit with status 1 Move some files about Create some directories and move files to those directories Call a script to set up the network Install NIS (A.K.A. YP) Set up NIS Set up ability to share home directories Call a script to set up a Solaris Zone (virtual machine) Move a customised file over a system file Call a cleanup function

Now, if any part of that script goes wrong, the script will fail (it will because I’ve used the “set -e” option).  If the script exists half of the configuration is done, and if you try to run the script again, it will fail because some of the setup is already complete.

To get out of this horrible situation, I have used a trap.  If the script exists with an exit status of 1 or greater, or if it is interrupted by a HUP, INT, QUIT or TERM signal, it will be trapped and run a function called “abort”.  Here is my trap statement:

trap ‘abort’ 1 2 3 15

My “about” method looks something like this:

abort(){

echo “ABORTING!  Please take note of any warnings!” #If pkg-manage exists, rename it to pkg if [[ -a /usr/bin/pkg-manage ]]; then rm /usr/bin/pkg mv /usr/bin/pkg-manage /usr/bin/pkg fi #Call cleanup to clear away temp files cleanup #Remove contents of /var/vaes rm -rf /var/vaes #Remove config file rm $CONFIG_FILE #Uninstall NIS & related things pkg uninstall SUNWyp domainname "" rm /etc/defaultdomain cp /etc/nsswitch.files /etc/nsswitch.conf zfs set sharenfs=off rpool/export/home cat /etc/auto_home | grep -v "`hostname`:/export/home" > /tmp/auto_home mv /tmp/auto_home /etc/auto_home #Delete zones dir POOL=`zfs list | awk '{ print $1 }' | grep "export" | sed 's/([a-zA-Z]*)/.*/1/g' | head -1` zfs destroy -Rf $POOL/export/vaes-zones #Undo zones bash /tmp/zone_setup.sh abort #Undo networking bash /tmp/network_setup.sh abort }

By using the trap statement, you can clean up after your failed script in the majority of cases.  A word of warning though: You cannot trap the KILL (or 9) signal, if your script fails because it has received a KILL signal, it will just exit.

Robust Bash Scripts - Part Three

This is part of a series of posts which should aid anyone writing Bash scripts to make them more robust.

This post offers some advice on how to Bash script defensively.

Be Prepared is the old Scout motto, and never is it more relevant then when scripting in Bash (or any language come to think of it).

Missing files and directories

If you are going to be working with files and directories in Bash (and you probably will be 99.9% of the time) you should test to see if they exist.

You can do a very generic test like this:

MYFILE=/tmp/test

if [[ -e $MYFILE ]]; then #Do something else #Do something else #Perhaps create file/directory fi

The “-e” will test for the existence of ANY file, including directories.  You will probably be better off using “-d” for directories though, and “-f” for regular files, see the man page for more information about test (use the command “man test” or “help test”).

If you are creating a directory somewhere like “/opt/local/share/lib/foo/bar”, you should either test for the existence of each folder, or you can simple used the “-p” flag on the “mkdir” command, this will create any folders that aren’t created.

Spaces

Sometimes your script will deal with text that has spaces in it, and you need to be prepared for that, even if you are sure it will never need to handle spaces, there will always be one occasion somewhere down the line.

If we take this example:

for EACH in $@;

do echo $EACH done

Say the input to the script reads something like this: bash myscript.sh hashbang0 “Ben Lavery” Aberystwyth The for loop shown previously would print out the output like this: hashbang0 Ben Lavery Aberystwyth

If we quote the $@ like this:

for EACH in "$@";

do echo $EACH done

The output now becomes:

hashbang0 Ben Lavery Aberystwyth

Failing Gracefully

If you are updating a lot of files in a directory, what happens if your script fails halfway through?  Half of your files are unmodified, and the other half are modified.  This could be disastrous, especially in something like a directory full of web documents.

One solution is to copy the contents of the folder before you start working on them, do the work, move the copied files over the old files.  Something a bit like this:

cp -a /tmp/mydir /tmp/mydir-temp

#modify files in /tmp/mydir-temp mv /tmp/mydir /tmp/mydir-BAK #backup mv /tmp/mydir-temp /tmp/mydir

As long as you do the relevant testing and exit the script if anything goes wrong, the script will never overwrite /tmp/mydir, and if it does, you have a backup!

Robust Bash Scripts - Part Two

This is part of a series of posts which should aid anyone writing Bash scripts to make them more robust.

This post shows you how to use Bash’s built-in error checking.

Sometimes you might write a script where one line of the script may fail, this failure could cause the rest of your script to fail horribly!

For example, you have two scripts: One which sets up the network on a machine, and another which sets up a naming service based on details given in the network script.  The second script calls the network setup script before it sets up the naming service.

If the network setup fails for some reason, the script shouldn’t attempt to set up the naming service as something has gone wrong.

You might try something like this:

bash setup_network.sh

if [ “$?”-ne 0]; then echo “command failed” exit 1 fi #start setting up naming service

Although this is fine, what happends if you forget to add it?  Or assume that the network will always be set up so decide not to add it?  If the network doesn’t get setup, you’ll be in a bit of a pickle!

Bash provides its own, built-in, error checking for you in the form of the “errexit” option, or “set -e”.  If a command fails in a script with this option enabled, the script will automatically stop with an error code 1.

If you need to turn off error checking (and you may need to if you are using something like grep which may fail, but you know you’ve handled it), then you could do this:

command1

command2 set +e command_that-may-fail set -e command4

“set +e” will turn off error checking.

In a later post I will look at the `trap` statement to catch the error this option throws.

Robust Bash Scripts - Part One

This is part of a series of posts which should aid anyone writing Bash scripts to make them more robust.

This post shows you how you can guard against unset variables.

Bash doesn’t provide any sort of checking for unset variables by default.  You may have something similar to the following piece of code: #!/bin/bash #Script is called delete.sh DEL=$1 rm -rf ~/$DEL You would use it similar to this: `bash delete.sh tmp`, which would delete ~/tmp.  Imagine if you forgot to add the “tmp” on the end…POOF!  That’s your home directory gone!

You should get into the habit of using the Bash option “nounset”, otherwise known as “set -u”.  If you put that at the top of your script and run the command without any arguments, you’ll get something like this:

$ bash delete.sh

delete.sh: line 3: $1: unbound variable

This is somewhat similar to a runtime error in Python or Perl and could be very annoying, but at least you’ve saved your home directory, or another critical location on your computer!

As a side note, if “set -u” or “set -o nounset” is specified at the top of the script and the script exists because of an error, it will exit with an exit status of 1.  In a later post I will explain how to use the `trap` statement to catch this error.

May the Fourth be with you!

Happy Star Wars day everybody!  May the Fourth be with you!

Exhibition in London

Faye and some others from the Aberystwyth university are putting on an exhibition in London between 2nd April and 16th April.  The exhibition is taking place in the Proud Central Gallery (32 John Adam Street - WC2N 6BP), just off the Strand and literally a minutes walk from Charing Cross.

The exhibition is the display of 19 artists work, all in the medium of photography.  It’s well worth a visit, especially as it’s FREE to get in!  Support the artists by buying a catalogue (a bargain at £4) and if you see a piece of work you like, please buy it!

Next generation Sun Ray hardware is out!

Oracle have uncovered a new Sun Ray (ultra) thin client recently.  They seem to have been quite quiet as I haven’t heard any news about it.

The client is called the Oracle Sun Ray 3 Plus and boasts the following features:

  • Low power consumption (typically around 14W)
  • 4 x USB 2 ports
  • Gigabit Ethernet
  • SFP (all Form Pluggable) fibre optic module for use with 1000 SX, 1000 LX, or 100 FX fibre optic connectors
  • 2 x DVI connectors
  • 2560 x 1600 resolution for a single display or 5120 x 1600 for two displays
  • Smart card reader One big jump from the older Sun Ray 2 hardware is the gigabit Ethernet, with it’s greater transfer speeds, this makes the four USB 2 ports possible.  Clients around the globe will be very happy with that.

The standard Sun Ray 2 supported a 1600 x 1200 resolution, even the Sun Ray 2FS client only supported 1920 x 1200 or 3840 x 1200 resolutions.  The Sun Ray 3 plus has a massively higher resolution with a single monitor supporting 2560 x 1600 or a dual monitor setup supporting 5120 x 1600 resolution thanks to the more powerful graphics card and the two DVI ports.

The fibre connectors are nothing new, they were present on the old Sun Ray 2FS and, of course, the standard smart card reader.

The only negative thing I have to say about this new hardware is that it takes up around 10W more than it’s predecessor.

I like the fact that Oracle have released this, it shows that they are committed (at least in the immediate future) to Sun Ray and the Sun Ray software.  I’d love to see one of these things, they should really help push Sun Rays into the workplace as a very viable alternative to those lumps of metal on everyones desks!

My Mac knows about the iPad

I was just browsing through the /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources folder on my MacBook, just to see what was there, and I suddenly spied the iPad icon:

I’m not sure what update brought this, over the past few weeks we’ve had the 10.6.3 update, though I suspect it was the recent iTunes 9.1 update, I’ll check with my Mac Pro back in Aber, I haven’t updated the iTunes on that in a couple of weeks…

Image is copyright Apple.

Weekend in London

I’ve spent my Easter weekend in London with Faye. She has a exhibition at the Proud Central Gallery (Open for the next two weeks!). I traveled from Aberystwyth to Faye’s home in Carmarthen on Friday, then from Carmarthen to Cardiff by car on Saturday, then Cardiff to London on the MegaBus. Saturday was the opening, and it went very well indeed!

I spent Sunday in London too, walking around shops and generally seeing the London sites. I still marvel at the London Underground, I think it is an awesome service, I love it! I’m also marvelling at the state of modern technology. I love that I can hook my phone up to my laptop for internet use, which is what I’m doing at the moment, whinging my way down some train tracks out of London Paddington while writing this blog post. I think its brilliant!

Anyhoo, off home to Melksham now. Will be seeing family and friends for the next week or so, as well as trying to tidy my dissertation code.

Dropbox

Ever wanted to store stuff online but couldn’t afford web hosting?  If so, Dropbox is your answer!

Dropbox offers both free and paid for storage online, the free storage has a limit of 2GB, whereas paid sotrage has a limits of 50GB and 100GB depending on how much you want to pay.

If you use the free option, you’ll start off with so much storage, and if you invite friends to join the sservice, Dropbox will add more storage to your account!

If you want to help me out, click here :)

Dropbox also integrates really nicely with Mac OS X, Linux and Windows, it feels just like another folder on your computer.  Drag and drop stuff into the folder (or cp/mv on the command line ;) ) and Dropbox will sync it to it’s online servers.  If you go to another computer that you’ve installed Dropbox onto, it’ll download any new content!  They also have a nice web interface incase you don’t have permission to install the software on your computer.  Lastly, Dropbox allows you to share files with other Dropbox users, which is really quite handy!