Home NAS & Server (5): Tweaks, Quick Tests, and Data Transfers

This is Part 5 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index

Once things are set and up and running, the first step really is to make all those tweaks and short tests that we want to do to ensure things are running as we want, and then migrating our existing data over to the new system.

Whilst prepping this particular post, I went back to try and find my references – particularly for the tweaks. As Murphy may well have predicted, I couldn’t find them. I’ll do my best on each point to either find them after the fact, or at least state where I saw them and explain them as best I remember, but I apologise for any I’ve missed or miscredited.

Limiting ARC

There’s a bunch of decent references for this on Solaris Internals ([1] and [2]) and some good threads covering how to do it on the zfs-discuss list.

There are a few reasons you might want to do this but it’s worth noting that the defaults in ZFS on Linux were changed from those Solaris used so that it defaults to 50% system memory – https://groups.google.com/a/zfsonlinux.org/forum/?fromgroups=#!searchin/zfs-discuss/arc/zfs-discuss/4iNT8aXouzg/UL6xf69HkjMJ

Additionally, this thread explains how you might make such changes, and introduces a couple of other options you may consider, particularly if you’re planning on running zvols (for block devices) and so on.

At this point I’m not making any of these ARC changes on either build until I’ve monitored how things pan out performance-wise, but they’re good to know about.

Disabling / Limiting Swap

Reference: https://groups.google.com/a/zfsonlinux.org/d/msg/zfs-discuss/PzBAABrZCb4/5Z-tscgBBaQJ amongst other places.

Anyway, the basic premise here is that “normal” swap is your enemy – and your weak link in the chain – in terms of data protection and checksumming in  a ZFS on Linux capacity.
Why? Well, the basis of it is down to the difficulties in easily running ZFS root under Linux. I haven’t tried to do this yet, but the difficulty comes in being much more careful on how you upgrade things where the root is concerned – especially if you want to keep the experience pain-free. Because of this, a lot of people will install root on “normal” drives, including swap. Once that’s done, if you ever use that swap space (to help avoid out-of-memory conditions) you’re sacrificing all the hardwork and additional money you put into ECC RAM – your swap has become a single point of failure.

You can, of course, create a small ZFS filesystem to use as your swap partition. Or have a large amount of RAM that (like in my position) you may not touch for a long time, but an easier method may be to just turn “swappiness” down to 0. I’ll let you reference the various pages about this yourselves but, effectively, this will all but disable swap unless the system really, really needs to use it (to avoid out-of-memory). The easiest way I found to do this (and make it permanent) was to add a file (swappiness.conf) in /etc/sysctl.d/:

~$ cat /etc/sysctl.d/swappiness.conf
# Only swap to avoid out-of-memory
vm.swappiness = 0

This will get enacted on the next reboot, or you can run sysctl -p /etc/sysctl.d/swappiness.conf to force a read of the options in the config file instantly. Similarly you can echo the “0” straight into /proc. The important thing is to make it persistent across reboots.

Copying Data from the QNAP

Going into this, I barely thought about the data migration. It seemed simple – rsync, right?

It made sense. The QNAP can act as a rsync server with ease (Administration Panel > Applications > Backup Server) and, once done, there was no problem installing rsync on the new NAS and connecting to it with:

# rsync -avz admin@nas:/share/Public/ISO/ /srv/multimedia/ISOs

A quick prompt for my QNAP admin password and it starts syncing, compressing the stream and maintaining attributes.

There was one major issue though – it took bloody ages. Seriously, a long time. Almost a day. And that was just for the music directory, which is only around 110GB in size (Films alone occupy more than 1TB). The average stats rsync reported at the end of the transfer happily told me it had averaged a mighty 3MB/second. Put simply, that was not the performance I was looking for.

I freaked out a bit. Raidz2 couldn’t really hit performance that badly, could it? None of the research I’d read suggested it would. I ran a dd to one of the newly created filesystems, just to check the disks were behaving alright. Performance was good. I didn’t write down the figure, but it was around 800MB/s – the disk layout isn’t the issue.
It shouldn’t be compression (lz4 is enabled by default across the pool) – all the supporting documentation suggests the overheads are minimal and, with modern hardware, there’s little reason not to enable it pool-wide, even if the gains are minimal for some pools.
Is it the network? I’ve got a fairly respectable Zyxel GS-2200 Gigabit switch that, if anything, is massively under-utilized. Besides, nothing else but my desktop was using the network at the time. It seemed unlikely that was the issue either.
I tested without the “-z” (compression) – no joy. I tested a transfer of some extra ISOs from my Windows desktop over SMB – I got a respectable 100MB/s out of it, even with the SMB overheads. Strange.

I researched a little bit and came across a couple of posts that pointed to SSH and the encryption overheads as being likely to slow things down. I doubted they were doing much to tax the new system but it seemed reasonable that they would potentially kill the TS-219P, which – as we covered earlier – is not exactly the meatiest of machines.

As my TV Shows transfer was still running (over the default method, SSH) I figured I’d try the points raised in those posts and mount the NAS locally over NFS and then rsync “locally” from the mountpoint to the ZFS volumes. Comparatively speaking, it flew out the door. Here’s a couple of quick rsync summaries:

rsync TV-Shows folder (SSH):
sent 55932 bytes received 674416624286 bytes 5175538.67 bytes/sec
total size is 674334091593 speedup is 1.00

rsync Live-Shows folder (NFS):
sent 18278740591 bytes received 660 bytes 54644966.37 bytes/sec
total size is 18276506379 speedup is 1.00

It may not be the most scientific of tests, but it’s around 10 times faster. Off the back of that I’m now confident enough to try and move the Films all in one go, with “-z” added back into the mix. Hopefully, it won’t still be running this time next year.

In transferring everything with rsync, I noticed another little issue I hadn’t noticed / been aware of before – a massive amount of “.@__thumb” directories. A quick search verified their source – the TwonkyMedia server running on the QNAP.

I don’t intend to use Twonky for DLNA on the new build, nor do I have any desire to keep a bunch of useless hidden folders lying around my filesystem. I knew I could get rid of them using find, but wasn’t entirely confident on my syntax. This page helped, and in the end I was able to do it comfortably with:

find /srv/multimedia/ -name ".@__thumb" -exec rm -r {} \;

It had soon whipped through even the more verbose directories (like the Music folder) and removed all the junk. As far as I can tell, no genuine files were harmed in the running of that command – nor can I find any reason why they should have been.

With the exception of the films currently transferring, the data migration has been painless and straight-forward. The slow speeds were obviously a concern but at least the result of that was application- / protocol- based rather than something more fundamental. Right now though, the basic data is in place and sitting pretty:

:~$ sudo zfs list
pool1 946G 4.26T 232K /pools/pool1
pool1/backup 28.8M 4.26T 198K /pools/pool1/backup
pool1/backup/backuppc 28.6M 4.26T 28.6M /var/lib/backuppc
pool1/home 239M 4.26T 198K /pools/pool1/home
pool1/home/dave 239M 4.26T 239M /srv/home/dave
pool1/multimedia 946G 4.26T 198K /pools/pool1/multimedia
pool1/multimedia/Audio 107G 4.26T 107G /srv/multimedia/Audio
pool1/multimedia/Books 267K 4.26T 267K /srv/multimedia/Books
pool1/multimedia/ISOs 2.54G 4.26T 2.54G /srv/multimedia/ISOs
pool1/multimedia/Photos 2.59G 4.26T 2.59G /srv/multimedia/Photos
pool1/multimedia/Video 834G 4.26T 834G /srv/multimedia/Video

Home NAS & Server (4): Setting up ZFS

This is Part 4 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index

EDIT – April 2013: Since originally writing this segment, ZFS on Linux has reached it’s first non-RC version – 0.6.1. With that release, a Debian Wheezy repositories are available that handle the install process. If upgrading from these instructions, the previous packages must be purged fully before installing from the repo (dpkg –purge zfs zfs-devel zfs-dracut zfs-modules zfs-modules-devel zfs-test spl spl-modules spl-modules-devel). Up-to-date Debian-specific instructions are at http://zfsonlinux.org/debian.html)

Fortunately, the good folks involved with ZFS on Linux have got both useful instructions on compiling your own packages along with a lovely Ubuntu PPA repository for both stable and daily releases. Conveniently the Lucid PPA works perfectly with Debian Squeeze, although I’ve settled on Wheezy for the HP ProLiant Microserver, so for that I’m building from source.

Compiling and installing ZFS

In testing (within a virtual environment) I followed the deb building advice for my Squeeze machine, and found the only thing I needed to do was run update-rc.d zfs defaults after install to ensure that the pool is automounted on boot. They were just as painless for Wheezy, and needed no trickery to automount the zpool on boot.
Using the PPA on Ubuntu I had no such concerns.
The PPA pages detail all that is really needed to add the repository to Ubuntu, and installation is as simple as aptitude install ubuntu-zfs but there were a couple of different steps needed for the Debian Squeeze system. As Wheezy shouldn’t be too far off become the new stable, I won’t spend any time on those steps – Google is your friend.

Once it’s finished installing and the kernel modules are compiled you should be able to run zfs and zpool commands.

Preparing zpools

I won’t be using hot swap / external loading drives on either of these builds, at least not to begin with, so having a relatively painless way to identify which disk is which is relatively important to me. To that end, I decided to use the vdev_id.conf file to allow me to assign human-readable names to all the installed disks. As detailed on the project page, this technique is really more useful in much larger environments to ensure you can quickly and easily identify separate controllers for greater redundancy in your pools. In my case it’s more so I can quickly and easily identify which disk is broken or misbehaving when that time comes. A quick cross-reference of the disk serial numbers before I inserted them into the bays with the device assignments once booted helped me confirm the correct PCI addresses. The naming I decided on was:

~ cat /etc/zfs/vdev_id.conf
# Custom by-path mapping for large JBOD configurations
# Bay numbering is from left-to-right (internal bays)
#<ID> <by-path name>
alias Bay-0 pci-0000:00:11.0-scsi-0:0:0:0
alias Bay-1 pci-0000:00:11.0-scsi-1:0:0:0
alias Bay-2 pci-0000:00:11.0-scsi-2:0:0:0
alias Bay-3 pci-0000:00:11.0-scsi-3:0:0:0

After doing the above, running udevadm trigger will cause the file to be read and the symlinks to the correct devices will appear in /dev/disk/by-vdev/. The benefits should become clearer once the pool has been created.

For reference, Bay-0 is my root drive – I’ve not opted for ZFS on root, nor any real resilience due to space constraints (I may work around this in the future). For the rackmount build – as mentioned – I’ll be looking at small 2.5″ drives mirrored using mdadm for the root files.

At this point, all that’s left to do is create the pool. As mentioned, Bay-0 is my root disk, so the disks I’ll be looking at using are Bay-1Bay-2, and Bay-3 – configured as raidz. To confirm things, I ran zpool create with the -n flag to verify what would be done. Once happy, the -n can be removed and the command run:

zpool create -f -o ashift=12 -O atime=off -m /pools/tank tank raidz Bay-1 Bay-2 Bay-3

A couple of notes on this:

  • I used -f because I received the following error without it:
    invalid vdev specification
    use '-f' to override the following errors:
    /dev/disk/zpool/Bay-1 does not contain an EFI label but it may contain partition
    information in the MBR.

    I’ve seen this error in my testing as well, and found a few threads on it, but nothing convincing. I confirmed with fdisk that the disks do not, in fact, contain any partition tables, and opted for -f to force past it.
  • ashift=12 is per recommendation (drives with 4k blocksizes, per http://zfsonlinux.org/faq.html#HowDoesZFSonLinuxHandlesAdvacedFormatDrives
  • atime=off as a default setting for the pool filesystems as I can’t see any real reason to enable it across them all.
  • -m allows me to specify a mountpoint – whilst I’ll only have one pool, I figured specifying a specific pools directory would be a good habit to form if I ever want to import other pools in the future. The directory must exist before this command is run.

Once created, things can be confirmed with zpool status:

~ zpool status
pool: tank
state: ONLINE
scan: none requested
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
Bay-1 ONLINE 0 0 0
Bay-2 ONLINE 0 0 0
Bay-3 ONLINE 0 0 0

At this point I’m ready to do pretty much what I want and can verify my pool is online and mounted with df.

Home NAS & Server (3): Hardware Choices

This is Part 3 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index

Once the software choices had been settled on, it’s easier to get a clearer picture on what I want to get from the hardware along with quite what spec I need.


Having initially wanted to make the new case fit into a 3U max space (which is what I happen to have free in my cabinet at the moment) I was getting a bit nervous when trying to find options. What I began to find was that the vast majority of the available options will easily fit in 2U or 3U, but most end up requiring 26″ depth, which I don’t have. The smaller cases were largely ruled out as they required boards that couldn’t take the amount of RAM I’d want, or didn’t have the requisite disk / expansion slots to cater for the storage side of things.

For a long time I was looking at the Antec rackmount cases (2U/3UEPS systems) but found them hard to source and I was concerned about getting tied in to specific (expensive to replace) PSUs. As time’s gone on it seems they’re hard to find because they’re being wound back, if my recent visit to the Antec website is anything to go on. I later shifted to looking at the Chenbro RM-22300 because it’s seriously cheap, and meets the form-factor requirements. As an additional bonus it also fits standard PSUs, although it looks a bit of a cludge in how its cabled. It’s big drawback is that whilst it would fit the bill today, it lacks room for any real expansion.

Looking back at things more recently, I’ve figured I can safely sacrifice my 2U shelf in order to make room for a 4U case. Whilst I don’t need anything that size right now, it pretty much allows for a full-sized desktop case which means normal PSU, lots of drive bays, and plenty of easy-to-facilitate expansion. Provided I get some fixed rails to support the weight, there’s no real reason not to look at it.
As I liked the look (and reputation) of them already, I went back to the Antec 4U22EPS650 option which comes with a PSU and has plenty of room and options. One nice point is it can potentially be expanded to have 3×5.25″ slots available on both sides of the rack, allowing for hot-swap bays to be added fairly painlessly. It’s easier to find than the other options and will easily address any future expansion concerns.

Hot Swap Drive Bays

Whilst not exactly a key requirement to be in place on the new build from the get-go, easily accessible drive bays is something I’d definitely like to have in place.

I’ve been looking at a few possibilities, but the one that still leads at the minute is on of the 4.35″ in 3×5.25″ options from IcyDock. Not being in a position to get one yet, I haven’t quite decided whether I want to go fully tool-less or not, but price will probably be the main deciding factor. Otherwise, they’ll all do SATA3 which is my only real concern.


As I’ve already covered, one thing I am keen for this box to do is cater for other projects beyond the scope of a “standard” NAS, including acting as a hypervisor and a PVR backend. So it could do with a bit of oomph in the CPU department, whilst at the same time being conscious of the fact it will be on most (if not all) hours of the day. Given that I’ll be using ZFS for my storage, it also needs to be (ideally) able to work with ECC memory. The more I looked at available desktop options, the more I came to realize that using one of the newer i3 / i5 / i7 Intel chips would mean sacrificing ECC or paying a lot more for enterprise boards – which I can’t justify at the minute. ASUS, on the other hand, offer consumer boards that support ECC merrily in their AMD line.
So I got looking to AMD CPUs. Looking at the stats, the energy efficient ones have considerable savings, but were hard to find in the UK. Not impossible though, and I managed to pick up a 615e on import. I just needed a Heatsink / Fan for it, which eBuyer helped with.


Having already come down on the side of an AMD CPU and ECC RAM, I was looking fairly firmly at Asus to help fulfil the motherboard requirement, and I don’t think they’ve disappointed. The M5A99X EVO ticks all the right boxes and caters for a good number of hard drives (6 using SATA 6Gb/s ports and 2 using 3Gb/s) along with ECC RAM support.
As with the CPU decision, power usage here won’t be as low as I might have wanted from a “pure NAS” box, but it will should have more than enough grunt for the additional tasks I want it to perform.


The MB can take up to 32GB ECC RAM, but I found 8GB ECC Unbuffered DIMMs hard to come by. All the main manufacturers’ sites have them, but finding them at suppliers was more of a challenge.

In the meantime then, I’ve opted to settle on 4x4GB DDR3 PC3-10600DIMMs from Kingston, KVR1333D3E9S/4G. The pricepoint is pretty good for my plans, and 16GB should give me plenty to work with.

Longer term, 32GB is the plan, using 4 8GB DIMMs, but I can expand to that when needed.

Hard Drives

At the same time as I first started thinking about doing this build, Western Digitial released their Red drives, aimed specifically at the SOHO NAS market. I’ll be honest, I don’t know a vast amount about the intricacies of hard drive technology – and I do appreciate that “Enterprise” / “Server” drives will always be more substantially tested than SOHO / Desktop equivalents – but in the reading I did around these, the implication seems to be that they’re worth a go. Firmware improvements, longer-than-normal warranty and being pitched (and tested) as always-on seems to make them a reasonable choice. I guess time will tell.

When I first set about this, I sort of default assumed I’d be looking at the 2TB drives as a price compromise between safety and resilience, either in a mirror or raidz (RAID-5, effectively) configuration – so tolerable of 1 disk failure. However, recent discussions haveencouraged me to look towards using the 3TB disks, but upping to raidz2 (double parity).

For the boot disk, I’ve opted to go with a single 2.5″ SATA2 drive from WD. I might eventually up this to a software mirrored RAID option, but as all the important details will be backed up to the ZPool anyway, downtime for a single disk failure isn’t the end of the world (at this point).

Side Note: The Other Build

As noted in passing earlier, I am also building a similar (but slightly lower spec / requirements) machine for my parents. The intention is this will serve both as a central store and backup machine along with serving as an off-site backup for myself using snapshot send / receive.
This machine wouldn’t need to do the extra stuff I want the rackmount box to do. It doesn’t need to PVR and it doesn’t really need to host VMs. It needs to hold data and run some relatively light services for the rest of the network.

Given the £100 cashback offer that HP are still running, I opted for a HP ProLiant N40L Microserver, replacing the 2GB RAM with 2 x 4GB ECC RAM modules and adding a StarTech PCI-E Dual-Port Network Card to add the additional ports the box will ultimately need. For disks, I used three Western Digital Reds, 2TB each, and kept the 250Gb that came installed in the system as a root drive.

Home NAS & Server (2): Software Choices

This is Part 2 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index

Given the scope of what I want the new build to do, it feels wise to decide what software I want to use to provide those capabilities, as it’s more than likely that some of those choices will effect the hardware specifics they need to run upon.

Looking back over the previous list I feel there’s a few key points to cover, namely Operating SystemFilesystem / RAID, Virtualization, and DVR. There’s probably more besides that I’ll think of later.

Operating System

It’ll be no surprise that the OS is going to be a flavour of GNU/Linux here (it’s sort of the point of the project), but I should probably justify both why, and which flavour I’ll be looking at using.
Using Linux is the obvious choice to my mind based on a number of factors, not least of which is familiarity (as a server platform) combined with my general preference for Open Source software. It’s cheaper too, of course, although to be honest that’s less of a major factor given that Home Server versions of Windows aren’t massively expensive these days. The thing is, I’ve never really liked Windows Server stuff, even now I’m in a position to have access to them through labs at work, they just feel clunky. Solaris / OpenSolaris / Illumos, too, I disregarded as I’m not as familiar with it and it, frankly, doesn’t feel as flexible as my experiences with Linux.
I considered BSD-based solutions which, whilst viable, I’m less sure I can get things like the PVR-backend functionality (and TV Tuners) working on it without considerable (additional) messing around. I’m going to try and get more familiar with BSD, but I can’t justify this build as being the time to do it – given the Swiss Army Knife approach.

In terms of the distribution, I originally came down on the side of Ubuntu 12.04 LTS Server Edition, with a view to moving to Debian 7.0 (“Wheezy”) when it’s released. However, with Wheezy’s release being so close, I figured it’s reasonable to start with it straight out of the blocks.
Debian’s what I’ve done most of my playing around with and I’m a big fan of APT for package management. However, I’m less confident of getting TV Tuners working with Squeeze, unless I start to draw more heavily on Backports – if I’m going to do that, I’d rather use Ubuntu LTS in the interim, and jump to Wheezy when it’s “new”.

Filesystem / RAID

When I set off down this path of building my own NAS I genuinely hadn’t given much thought to how I was going to handle this aspect. In my head, it was obvious – mdadm software RAID (I can’t really force any justification for using hardware RAID), most likely with LVM on top of it and most probably ext4 as the filesystem.

The more I got to thinking about it though, the more I got comparing some of my desired features with the kit I get to play with at work (NetApp filers of varying size and shape). Whilst I had previous exposure to snapshots through ZFS on Solaris, I never fully appreciated quite how powerful and useful they are. I like the idea of being able to do relatively easy, incremental backups to a remote location; I like the scalability of it (even if the scale is way beyond anything I’d need for a home NAS); and I like the extra steps it takes in checksumming and data integrity. Given the way ZFS handles disks, it effectively takes care of the software RAID side of things at the same time.

The downside is that ZFS isn’t really native on Linux (I’ll let Google help you, but fundamentally it’s a licensing issue). There’s a ZFS-FUSE project but as you might expect, all the reports suggest performance is sluggish to say the least. Additionally, I’ve already ruled out using Solaris / BSD as a base. There is, however, a native port maintained as ZFS on Linux which has a pretty good implementation working (albeit not quite with all the features of the very latest ZFS). I’ve been following the project for a while and there’s some healthy activity and, generally, I’ve been impressed.

To keep things native, I also looked at BTRFS which seems to be showing a massive amount of promise but, as of yet, doesn’t quite tick all the boxes to me on older / non-bleeding edge kernels. It’s something I definitely want to keep an eye on and test further though, as it seems to have the potential to surpass ZFS as time goes by, especially with the continued uncertainty after Oracle’s takeover of Sun and whatnot.

So, whilst it flies a little in the face of some recommendations from friends, I’m deciding to trust a collection of things I read on the internet and going with ZFS On Linux at this stage.
The key point for me was send / receive snapshots (on a relatively stable kernel version) – as I’m planning to build a similar device at my parent’s place, it painlessly addresses my off-site backup desire.


Given the way I seem to be wired, when I first considered having this build server as a platform for a few VMs (both for “production” use and for testing environments) I got way ahead of myself in considering whether running the host as dom0 for a Xen hypervisor, or using KVM. In both cases, I was heading down a road of working out whether running the hypervisor kernel would have knock-on effects on the systems desired output (NAS / PVR) and how to mitigate that.

Eventually, I realized I was being ridiculous. At most, I’m going to be running between 3 and 5 VMs near-constantly, with maybe a couple of extras fired up for lab purposes. None of them are going to be particularly resource-intensive, nor should they be under high load. As much as anything, they exist to provide some sandboxing and allow me to mess with config management and other tricks and tips that can ultimately be useful and applicable in a work environment.
Before anyway chooses to state the obvious – I can appreciate that Xen / KVM would be pretty good skills to have, but shoe-horning them into my domestic NAS environment strikes me as overkill (and if it *strikes* me as overkill, it *definitely* is overkill).

In the end I think I’ve setted on using VirtualBox, at least in the interim, for a few reasons. I’ve got some experience with it already in headless mode (albeit, not enough); it provides enough of the passthrough features that make it versatile-enough for my needs (I run my current labs in VirtualBox on my desktop); decent management tools exist; it can do some fun stuff to aid the integration with the neater features of ZFS; from what I can gather, the key limitations really only manifest themselves at a business-level, rather than home-use.


As has been hinted at previously, I really have no firm alternatives to MythTV for this purpose.
I first saw MythTV in action first-hand at a LUGRadio meetup many moons ago and I was wowed. Whilst I don’t doubt that there are other alternatives available (most, to my knowledge, need Windaz) – and certainly others that are quicker to configure – the sheer scope of what MythTV has achieved appeals to me massively. And I want to be – in some way – involved.

But, I’ll be honest, DVR had taken a bit of a backseat for me in short-term plans due to the difficult integration with XBMC but then these two posts ([1] and [2]) got me excited about it again. The only difficult part will be choosing a decent, supported tuner.

Home NAS & Server (1): Need vs. Want

This is Part 1 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index

As may be apparent from the index, one key feature of this new build is that it needs to be a bit of a Swiss Army Knife. Whilst I’m well aware I could quite easily build a lower power, (probably) cheaper and quieter machine that would fulfil the primary requirement of providing more storage space, it wouldn’t be as capable of some of the other tasks I’d ultimately like this unit to do. For what it’s worth, if it was just about storage, I’d probably seriously consider another QNAP if I was going to go for a pre-bought option, I really have liked my TS-219P

Let’s start by covering what I currently use the QNAP for and what benefits it provides, so the bare minimum that this new build has to offer:

  1. Video storage – My own DVD rips and the like, centrally stored for access by XBMC installs
  2. Photo storage – Finally, I shifted them off my aging USB hard drive. Hopefully, the number will begin to increase again before too long
  3. Music storage – Accessed by various means, including a Subsonic server
  4. Miscellaneous files – a pretty lax (if I’m honest) and ad hoc approach towards backing up what I consider to be my “important” stuff
  5. Interoperability with *nix and Windows clients (feasibly this should stretch to Mac, although I have no immediate need for that. Pedants, hush)
  6. Decent (local) backup and resilience – alright, it’s just mirrored disks in a little box, but it’s been sufficient so far
  7. A stable platform – It’s been up almost a full year now and shows no sign of causing any stability issues. It gives me my files when I want them, and writes what I want to write when I want to write it (until recently, see “space consumption”)
So I have the minimal expectations. As you can see from the list there, the only thing I actually need to address the shortcomings of my current solution is more storage space, which I could easily do with either a larger (4-disk) QNAP device, or a mini- / micro-ATX board, 1U housing and a few disks slapped into a software array with mdadm.
But… as I’ve hinted at plenty of times already, I’m looking to kill multiple birds with one stone. I’ve already accepted that this is going to mean the system won’t be quite as lean on power consumption as I might have hoped for, but with the various advances in “green” solutions in more recent components I’m fairly confident this can be somewhat mitigated against. Worst-comes-to-worst, a lot of what I want the system to do doesn’t require it to be powered on all the time, so making allowances for that is an option.

But enough of that, what do I want this build to do and what do I need it to do?

Need Want
  • Fault-tolerant storage
  • Multiprotocol (Windows, Linux, Mac) file sharing
  • Usable capacity of 3TB or more (minimum 50% increase on current capacity)
  • Command-line access
  • UPS integration
  • Power-saving features (schedules, WOL)
  • Able to perform as a central backup platform for other systems
  • Stability
  • Rackmountable (max 3U, max depth 22″)
  • (relatively) Quiet
  • Expandable
  • Configurable with (moderate) ease
  • Hot-Swap data disks
  • Capable of acting as an iscsi target
  • Network Interface bonding
  • PVR / DVR backend
  • Able to host a few virtual machines simultaneously
  • Easy remote (off-site) backup options – ideally incremental

And that is just about that.

Linux-Based Home NAS & Server

Shamelessly inspired by this brilliant series of blog posts – http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/ and http://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/ – both of which contain a great many more details than I will likely cover.

Before moving up to Newcastle, I picked myself up a QNAP TS-219P with a couple of 2TB drives mirrored. At the time, I ummed and ahhed about paying a bit more for a four disk version from an expansion perspective, but couldn’t justify it for a few reasons:

  • Extra cost – I didn’t have any present *need* for the extra capacity, and there was a reasonable step-up in price for a single-function box I didn’t need the full capabilities of.
  • Format – I knew if I was going to get a 4-disk-or-bigger version I would – longer term – want something rack-mountable, but at the time I had no rack to put it in. Without that, a rack-mountable NAS would just be silly.

Now, two years on, a few things have happened. Firstly, I’ve moved into my own place, complete with Cat6 wiring to as many places as I felt it practical and a rack to boot. Secondly – and perhaps more importantly in the context of these posts – I’ve reached the limits of my QNAP 2-disk solution. With an increasing multimedia collection, a refuelled desire to start taking more pictures again, and additional space for making a concerted effort to centralize my other data, more space is going to be needed.

However, as the next few posts should reflect, there’s a few other things I want the system to do, so the considerations (and cost) expand a bit beyond the remit of “NAS”.


Setting up the Sheevaplug as a Weather Station

UPDATE 25/5/2011:

$ uptime
07:42:35 up 94 days, 14:00,  1 user,  load average: 0.74, 1.32, 0.68

Pretty happy with that so far!

A few months back I got hold of a Marvell Sheevaplug from NewIT with the express intention of it replacing the aged, noisy desktop PC with a more efficient, lower-power and quieter solution. It’s sole intention was to run the open source weather station solution Wview – a tidy little collection of daemons that records the weather statistics provided by a variety of different weather station hardware and then allows you to do a bunch of things to it (generate your own HTML templates, send the data to Wunderground, Weather for You, and so on) .
With that in mind, my key spec and concerns for the device were as follows:

  • Run Debian Squeeze
  • Fetch data from an Oregon Scientific WMR968 via a USB-to-Serial adapter
  • Record the data consistently using WView
  • Given the risk of the SD Card failing, regularly dump a backup of the data to a remote store
  • Take steps to minimise the effect of regular writes to the SD guard and so prolong its life

When I initially got the SheevaPlug from NewIT, it already did run Debian Lenny, and I started taking steps to stick things like Flashybrid on there. However, I was hasty, and it ended up breaking the install.
Given that I was going to have to do it all again, I thought I’d best document it, especially as I could only find the information I used spread out over 4 different websites (see References, bottom). As such, these details are pretty specific to the exact steps I took, rather than the more general (and detailed) information that can be found at those references.

PLEASE NOTE: My installation environment was onto a standard SheevaPlug (no ESATA port), installing from a TFTP server onto the SD Card, and making that SD Card bootable. If you’re planned install environment differs to that, use Martin’s site to see what you need to do differently.
For all codes and commands, I have used
#~ to indicate the commands are being input on a new line.

Continue reading “Setting up the Sheevaplug as a Weather Station”

HOWTO: Automating Bridges and TUN/TAP

This isn’t ground-breaking stuff by any means, it’s more just a simple reminder for myself about how I did certain things in order to get a network bridge set up under Ubuntu 8.04, and to create a Tap connection that I could then use in VirtualBox to let routes and all that shiny stuff work. It doesn’t explain things fully (I don’t understand it), but it does cover what I did, hopefully step by step.
This only made sense thanks to the following pages:

1. https://help.ubuntu.com/community/VirtualBox#Create%20A%20Bridge
2. http://ubuntuforums.org/showthread.php?t=830777
3. http://ubuntuforums.org/showthread.php?t=752127

Anyway, let’s begin.

Does it Work…?

First step along this Rocky Road to Near-Fail was to follow the useful advice in Link 1 above, and thus making sure that creating a bridge and activating it and the TUN/TAP actually worked. It did. From that link, I did the following:

~$ sudo aptitude install bridge-utils uml-utilities

This installs the pre-requisite applications to do the fun stuff.
The second point depends on your viewpoint, but it’s probably worth backing up your current /etc/network/interfaces file in case you manage to break something:

~$ sudo cp /etc/network/interfaces /etc/network/interfaces.good

Obviously, what you call and where you place the backup is up to you. Just make sure it’s something you remember later.

Now for preparing the bridge itself. Fun:

~$ sudo tunctl -t tap1 -u USERNAME
~$ sudo chown root.vboxusers /dev/net/tun
~$ sudo chmod g+rw /dev/net/tun

Next up, we need to edit another file, apparently to help make permissions persist after reboots. The file we need to edit is /etc/udev/rules.d/20-names.rules
Again, we need to edit this as root, so from the terminal:

~$ sudo [$editor_du_jour] /etc/udev/rules.d/20-names.rules

And then at the end of that file, find the following line:
KERNEL=="tun", NAME="net/%k"
And add the following to make it look like this:
KERNEL=="tun", NAME="net/%k", GROUP="vboxusers", MODE="0660"

Take whichever process your editor takes for saving and closing that.

Now we can create the bridge itself:

~$ sudo brctl addbr br0

Now put the network interface into promiscuous mode, add it to the bridge, and set the Bridge to DHCP (if you are using DHCP, if not, ignore these and see the next statement):

~$ sudo ifconfig eth0 promisc
~$ sudo brctl addif br0 eth0
~$ sudo dhclient br0

If you are NOT using DHCP, and have a STATIC IP, follow this example:
~$ sudo ifconfig br0 netmask
~$ sudo route add default gw br0

(Obviously, replace the IP, Netmask, and Gateway IPs with your own…)

Now, simply add the tap1 device to the bridge and bring up the interface:

~$ sudo brctl addif br0 tap1
~$ sudo ifconfig tap1 up

Last thing I did was just to run ifconfig to double check everything that should be there is there. You should have the Bridge (br0) with your IP Address, the physical interface (eth0) set promiscuously, and the TAP, tap1.

Opening up VirtualBox and change the appropriate network settings for your VirtualMachine to point to the new tap device (in my case, tap1). First step in that is to change the ‘Attached To’ drop-down to point to ‘Host Interface’.

Screenshot of the Settings

Starting the VirtualMachine now should be effortless, and when it starts up (and you add them), the same routes you’ve been using should work just fine… so ping, ping away!

Making it Permanent

The initial instructions I was hoping to follow from Link 1 didn’t work out all too well for me, so I was back trying to work out exactly where I could fix it. Thankfully, SpaceTeddy on the Ubuntu forums was able to point me in the right direction of some useful hints he’d written.
In the end, I did the following.
First step is to go back and edit /etc/network/interfaces with your preferred Text Editor. You need to be root to do this. In there, you are replacing your current Physical Interface settings for the bridge, or, in my case, swapping out eth0 for br0. Then you are adding a rule to tell the bridge it using your physical interface (eth0). Finally you are adding the stuff that brings eth0 up as promiscuous. It should look like this:

auto br0
iface br0 inet static
bridge_ports eth1
auto eth1
iface eth1 inet manual
up ifconfig $IFACE up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down

It’s probably worthwhile noting that you SHOULD NOT remove the references to the Loopback Interface (lo), but do make sure any other references to your physical interface are commented out, or plain old deleted – you made a backup anyway, right?

After doing that, the only thing left to include is finding a way to bring the TAP interface up on startup. The other guides do mention ways to do it through /etc/network/interfaces but they didn’t work for me – I still don’t know why.

Instead, I just added the commands to /etc/rc.local, along with the routes I need to bring up everytime I startup. This was as simple a case as opening up the file in my preferred text editor (again, sudo is needed) and adding the following:

tunctl -t tap1 -u MyUser
brctl addif br0 tap1
ifconfig tap1 up

exit 0

Make sure to keep the ‘exit 0’ at the end of that file – it seems to work.

And that massively over-lengthy block of text is all that you need to do. I will try and refine this at some point but, this works for me and seems easy enough to follow if I need to remind myself what I did again.

Glastonbury Festival 2008

Well, it’s been a good day or so since we made it back to the real world after 5 days down in deepest Somerset to attend Glastonbury Festival 2008, so I thought it was about time I set about trying to actually write about it. This is that attempt.

In the Beginning

Attending the Festival as a group was John, Mike, Owen, Duncan, and myself. Duncan would be joining us on the Thursday, whereas the rest of us drove down on Wednesday. The journey down was uneventful, but we had the vehicle packed to the rafters with bags, tents, booze, and Pot Noodles. There really was little else.

The Music

So, this is the selection of artists I actually managed to see live whilst I was there. Of course, the sad thing is that I will miss some artists out, particularly those that I heard in passing or caught their sets midway through on the Wednesday or the Thursday, where no published program seemed to exist to help me find out the names of some of the funky musicians I listened to.
More or less, this list is in chronological order, starting on Thursday with The Levellers, and finishing with The Verve on Sunday night. You can fill in the other times and days yourself, or just trust me when I say they follow through the weekend…

Unfortunately, I didn’t get to see everybody I’d wanted to see, but I did get a fair old whack at things.

The Levellers

[last.fm linkage]
Various signs around the place on Wednesday and during the day Thursday had made it fairly clear that the Levellers were performing, and I decided pretty early on that I had to go see them. I don’t know why I thought that, because I’d only ever heard one track by them (knowingly) pre-Festival, and that was Just the One. But I went anyway, and plodded down to the Left Field tent in the pissing rain to try and get inside and be remotely dry whilst watching and listening. The place was full to the rafters.

As to the performance. Personally, i thought it was brilliant. Everybody loved it, lots of singing along, the sounds quality was good and the atmosphere was great – a really solid band to unofficially kick things off. Shame I got piss wet through in the process, but that’s what it’s all about I guess!

Kate Nash

[last.fm linkage]
Mornings soon proved to be not the best of times for our merry little crew, be it because we were sleeping, recovering, or just plain ol’ procrastinating instead of going out there and listening to music. Thanks to that, we arrived pretty late to see the last couple of songs Kate Nash was performing.

Kate Nash tends to get quite a lot of flack, or so it seems to me anyway and, if I’m honest, I think I was the only one of us who quite fancied going to see what she was like live, which probably doesn’t score me any points on the heterosexual scale. Bugger.
Still, I have to say what I saw at the end of the set was good, if you like the stuff she does, which I do. It’s entertaining, easy to listen to, and amusing, which is just fine in my book.

The Subways

[last.fm linkage]
To be honest, I hadn’t listened to much Subways stuff before the festival and wouldn’t have initially rushed anywhere to see them. I have the first album lying around somewhere and it occasionally gets a listen to and does the job well. However, they followed Kate Nash on the Pyramid Stage so we figured, why the hell not? There was nothing else particularly pressing that I wanted to see in the morning anyway.

As with what I saw from Kate Nash, I enjoyed it. They seemed to have a fairly solid stage presence and were fairly interesting to listen to. I only recognized a couple of songs from the first album, and wasn’t entirely sold on all the new material they tried out, but it was fun. Would see again (at a Festival… probably wouldn’t go out of my way to buy tickets).

Hobo Jones and the Junkyard Dogs

[last.fm linkage]
Fecking brilliant. John summed it up afterwards by commenting that this sort of thing is exactly what Glastonbury should be about- variation and random fun.

The atmosphere was brilliant, the performance was amusing and captivating and generally to a pretty good quality considering the washboard was in action. The three of them did brilliantly and genuinely seemed to convey an almost humble attitude to being there. They may have opened by saying it was a joint effort between the audience and them, but it really did seem that way as it went on – every single person in the room seemed to become a real part of the gig. Simply brilliant, and I hope to see these guys again at a future point. Try and listen to some of their stuff, it’s fun! 🙂

Alabama 3

[last.fm linkage]
Over to the Jazz World stage, this was another artist I really didn’t know very much about (except, of course, the Soprano’s Theme 🙂 ). They’re fascinating. In a really weird sort of way. I really can’t even begin try and describe them, as I wouldn’t know where to start. The vocals are a strange mixture but they seem to work and there’s something completely unavoidable about how cool and, dare I say it, funky the whole band is. Genuinely interesting.

Candi Staton

[last.fm linkage]
Everyone knows and loves at least one Candi Staton song, even if they don’t know her by name. And, arguably, this was one of the best sets I saw over the whole weekend.

Simply put, she’s brilliant. Superb voice, great stage presence, and everyone loves the music. She belted out some absolute classics, seemed to hit every note just the way she wanted to, and seemed to be having  a great time being there. If you didn’t see her, you missed out, regardless of whatever else you saw in her place. Sorry, you just lost the game.

Fun Lovin’ Criminals

[last.fm linkage]
The start of a crazy evening.
I’ve seen Fun Lovin’ Criminals before, in Lancaster, and thought they were great then. This time, they also didn’t disappoint, although I had a much more relaxed approach to watching them – that being that I was wandering around the gig on my own, too drunk for my own good, and talking to strangers (see ‘Reckless Abandon’, below for more). So in many respects, they were background music. But it’s great background music. This set also served to remind me that I should listen to them a bit more often. Tasty.

Sinead O’Connor

[last.fm linkage]
I was pretty excited at the prospect of being able to see Sinead O’Connor live in the run up to the festival, and so was a little disappointed with hindsight to admit that I didn’t actually see the show.
I was there – please don’t think I skipped it all together – but I just heard it again as background music whilst I was talking to two very nice people from the Czech Republic. Good times.

What I heard was good though, although it would be hard to claim she has a bad voice and, as with FLC, it was really nice to have good music going on around you whilst you could sit and talk to strangers, embarrassing yourself along the way. Lovely jubbly. I also failed to spot if she still had a shaved head, so anyone who can help confirm / deny that it would be much appreciated.

Kings of Leon

[last.fm linkage]
I’ve enjoyed listening to the Kings of Leon ever since I first heard Molly’s Chambers, which always struck me as fairly fun and easy to listen to. However, I will quite happily take the points made by both John and Mike that, relatively speaking, they are quite a boring band and don’t bring anything new to the table. That doesn’t bother me though. One thing Glastonbury reaffirmed for me is that I’ve largely got past worrying about where the music’s come from, what it means, what it brings to the table, or whether its overrated, ‘sold out’, or any other such phrases. Of course, if a songs got an emotional background, pushes boundaries and has something I can relate to in it then it will be preferred, but it doesn’t stop me just enjoying listening to things non-committally.

And so it was with Kings of Leon. I knew when I saw them announced that they were one of the few acts I would feel I _had_ to see whilst at Glasto, purely because what I’ve heard recorded I’ve enjoyed and – just as with so many other acts at the Festival – I really wanted to see what they could do live.

Honest verdict? I was disappointed. I enjoyed the songs, but I felt as a live act they were simply dull. I think I was expecting a bit more stage presence, and probably beer. I wanted rock and roll but all I got was this lousy CD-quality experience.
That’s not to say they were necessarily bad though. I did enjoy what was played and had a little solo boogie all to myself in my slightly spaced out state but, in future, I wouldn’t be overly bothered whether I saw them or went to see something else.

Pete Doherty

[last.fm linkage]
Following the end of Kings of Leon, I staggered up towards The Park to see the end of Pete Doherty, seeing as he’s another artist I really haven’t heard much from. I literally arrived right towards the end, but from what I did see and hear, he did a good job. Well played that man.

James Blunt

[last.fm linkage]
No one will believe me, but this was largely accidental. We lost Owen, and I had to hold the fort whilst the search parties went their separate ways. Turns out it meant I got to sit / sleep through this. Everyone there seemed to love it, good for them. Have to admit I was strongly apathetic, letting this rank as my biggest regret / wasted time moment of the weekend. It’s not that he’s necessarily bad, he’s just not really anything I’m into.

The Raconteurs

[last.fm linkage]
I dozed a little bit for this as well, but I thought they did a pretty good job overall. I knew hardly anything about them before sitting through it, and I probably would see them again given the chance. Not much more to say though really.

Manu Chao

[last.fm linkage]
Another artist I was determined to ensure I caught at least once during the festival, we decided to stick at the Pyramid Stage and see Manu Chao there. They didn’t disappoint. Energetic, upbeat, and all under the late afternoon sun. It couldn’t have been better.

A great mix of their songs, some new, some old, all timeless, it was everything I would have hoped for from a Manu Chao gig. Would definitely try to see them again.

Amy Winehouse

[last.fm linkage]
Not really a lot to report on this one that would be classed as ‘new news’, seeing as how everybody seemed to be talking about it and the ‘Punch up’ incident since the Festival. However, I thoroughly bloody enjoyed it. She was incoherent at times (could have been the sound levels). She was tottering about looking rather unstable. And the dress wasn’t the most flattering. But fuck it. She also has a brilliant voice and got stronger and stronger as the show went on. If newspaper reports are to be believed, and if she really is ‘ruining herself’ then, yes, of course it’s a shame and a sad story but, at the same time, if she’s having fun and making good music then… meh… maybe I’m just being selfish, but I actually quite enjoy seeing musicians embrace the rock and roll spirit every now and then. They’re only human, afterall. Why should their mistakes and life choices be plastered all over the place? And why should it bother us from a music point of view? To me, it shouldn’t.

So, Miss Winehouse, well done on the performance. I salute you.


[last.fm linkage]
It’s been spoken about enough, but the intro was brilliant, and showed a solid sense of humour. To then open up with 99 Problems was something I knew and appreciated. Good song.

Whilst I didn’t watch the entire set, the first half I watched was good. It was entertaining, had a decent variety of samples that brought back some songs I hadn’t heard for a long time, and it brought variety and controversy to the floor. I certainly hope there’s hip hop at next year’s Festival, even if it doesn’t headline. It brings it back to being a Festival in celebration of music, in all its forms.

Massive Attack

[last.fm linkage]
Given that the timetables clashed between Massive Attack and Jay-Z, I was only able to catch the last bit of Massive Attack before retiring for sleep. From what I saw, and from what I heard others say, it was epic. Great use of lasers as well. A little part of me is gutted I didn’t see it all, but then I was glad I got to see Jay-Z.

Neil Diamond

[last.fm linkage]
There’s very little that can be said against this mighty fine individual. In fact, if I had to nitpick, it would be because I thought his acknowledgement of the applause he received was a little repetitive – there’s only so many times I can hear “thank you so much” and hearing it after every song got a little boring.

Still, it was actually a lot better than I’d hoped it would be, with highlights clearly including Sweet Caroline and, for me at least, Pretty Amazing Grace since I’ve enjoyed the song immensely since I first heard it (repeatedly) on the Radio when it was released. It sounds just as good live.

Good clean fun would be a good way to describe the set. The afternoon sun made it very continental in it’s party atmosphere. Wild.


[last.fm linkage]
Yet another band I really didn’t know enough about / of before going to the Festival, I was very pleasantly surprised at quite how much I liked them. Granted, I dozed through a good portion of the set, but the music clearly wasn’t so offensive I couldn’t sleep to it. Great voice, and really quite interesting, I can imagine I’ll listen to more of this stuff before too long. Good job.

Leonard Cohen

[last.fm linkage]
The one artist I feel I can write an unhealthy chunk of text about.

Without doubt, this was the single greatest performance of the weekend (in my humble opinion), even if I wish to put all the weight of that achievement onto one song – Hallelujah. Simply stunning. Words don’t even come close. I am being completely honest when I say I don’t think I have ever seen any other live performance that has simply knocked me for six on an emotional scale as watching Mr Cohen belt out his version during this set. It actually makes the original album version seem rather disappointing by comparison, and it’s a song I absolutely adore.
As it happens, I had also been listening to Jeff Buckley’s version of the song a few too many times to be healthy in the weeks before the Festival, and that had always managed to intrigue me, but by comparison, it can’t do anything like Cohen’s version can. It’s husky, it’s candid, it’s bare bones, and it’s brilliant. It seemed to get possibly the greatest applause that I witnessed during the weekend and, if everybody was doing as I was, it was automatic. I could think of nothing else but applause. I couldn’t have shouted anything out. It genuinely hit me like a bottle of wine.

Maybe it was because I was still in a mildly fragile state from the previous few days, but it brought me bloody close to crying. Sad? Probably, but it’s the truth. Whether it was the combination of the music and the setting sun, with the added mix of having the entire crowd hanging off his every word with adoration or not I really don’t know. But it was special. I’ve linked a mobile phone video that somebody put on YouTube below. Check it out if you like. It’s rather good.

In many ways, it was a shame that Hallelujah came in the middle of the set because, as Mike said, it meant that the next few songs almost felt disappointing, but in so many ways I really don’t think Cohen cared, and he definitely kept everyone on board and interested as he continued through the rest of his set.

If you ever get the chance to see Cohen live, do it. Don’t think twice about it. Don’t wait. Just do it. You probably won’t get many chances. If he’s performing at a Festival near you and there are tickets left, buy them, and go. You will not regret it. It’s a statement I’m probably going to grow up to hate myself for saying, as it sounds a bit weak, but he’s an amazing performer, coming across as very commanding over his domain whilst at the same time seeming genuinely humble and honoured to be so adored. Plus, seeing as all his songs are largely, and primarily poems, I would be very surprised if he ever performs the same song quite the same on any two occasions. Brilliant.

Go watch the video: http://uk.youtube.com/watch?v=_tPQQLMXmQ4

The Verve

[last.fm linkage]
Probably the one band I was most excited about pre-Festival. The performance itself has no doubt been well documented. I thoroughly enjoyed it, especially Bitter Sweet Symphony. Nearly returned me to tears again. I believe it brought Owen closer.

Seriously good show though. Very impressive, and clearly appreciated by all who attended. They were every bit as good as you would expect them to be, and deserve any and all praise they receive. Solid performance.

Reckless Abandon

Friday night was Crazy Friday. In fact, Friday in general was Crazy Friday. But it was brilliant, and full of, you guessed it, reckless abandon.

I did everything my mother always told me I shouldn’t do:

  • I wandered around on my own in a strange new place
  • I drank excessively
  • I spoke to (and accepted gifts from) complete strangers
  • I applied the ‘cocktail’ methodology to my stimulants of choice
  • I picked my nose

With hindsight, it was foolish. It was also bloody brilliant, and something that I’d never done before. That’s not an encouragement though.

Anyway, where to begin? Well, my guess is it started with that Guinness at an ungodly hour of the morning. In so many ways it was a wrong choice, yet it seemed so right at the time. Lovely stuff. Then there could be the Hot Spiced Cider. After all, it’s only apple juice fundamentally, so we were all more than entitled to give it a whirl. Invigorating in and off itself.

As I remember it, at some point after this myself and John returned to ‘base camp’ to get some booze and lunch, by which I mean a Pot Noodle and the remains of the first box of red, red wine. Plodded on down towards The Other Stage to meet up with Michael and co, at which point the cave suit was on (it had started spitting again). Here came the first break, as the rest of them were off to see… someone… can’t remember who, and I headed back to Jazz World for the Alabama 3, ‘bag’ of wine now in hand (the box disintegrated in the rain). According to the strangers I was talking to, it looked quite suspicious, me being in, effectively, a boiler suit with a transparent bag in hand containing an unidentified red liquid. I’m sure they loved it really though.

At the end of the Alabama 3 set,which I enjoyed thoroughly, I latched onto some unsuspecting Welsh folk to chat about my dodgy appearance and why I was on my own (“No, really, I _do_ have friends, they’re just… erm… somewhere else…” 🙁 ). Just before Candi Staton came on stage, Mike and John found me (apparently, my bright yellow suit stood out… *shrugs*) and we made our way to the rest of them to watch – and subsequently boogie a little to – Candi Staton.
Once more, we got talking to strangers. A couple of fellows from down south, and a considerably older gentleman I’m led to believe was not with them, but who was, certainly, loving life. It was a solid hour of banter, and I thank them for that.

After that, I believe myself and John went back to the tent again, or maybe a few more of us did, to replenish food levels (with Noodles) and to grab another beer. I was to be the only one who wanted to see Fun Lovin’ Criminals over the other options so I set off back down there on my own, this time replacing my sun hat with the Indian Headdress. I could probably cite this as the defining moment of the evening.

On the way from The Other Stage to Jazz World, I stopped for another Cider, and shook many strangers hands, not to mention uttering the famous ‘How’ whilst raising my right hand a good few times. The concept made me giggle.

Arriving at FLC, I decided it was too busy for me to be bothered to get near the front, so I wandered aimlessly (and drunkenly) towards the edge of the sound stage. It was hear that I latched onto my second group of the evening – a jovial bunch of guys from Anfield (of course, I played the dubious card and admitted I was a United fan – not something I would have done sober). They were a really nice bunch and seemed to appreciate my odd attire. They offered me a beer which, for a Carling, settled surprisingly well, and soon introduced me to Gary, which I mistakenly thought referred to Green when they first mentioned him. Apparently, I was wrong, and I can only guess his name should be spelt Gar-E. But that’s just a best guess, the truth is, worryingly, that I have no idea. I know it wasn’t negative, and it made me eager to talk to strangers but beyond that, I’m stumped. The Gar-E suggestion came from Mike.

Anyway, the Jolly Scousers soon departed, and I was back to embarrassing myself in front of another group who, I think, were from London way and were only to eager to provide a screen for me to piss behind – a fact I had forgotten until a couple of days later – I know, I’m an embarrassment to my family. 😉

After chatting for a while, I realized that I was risking missing Sinead O’Connor, so I made my apologies and left. On the way I got talking to a delightful couple that were heading towards The Other Stage who decided to share some poppers with me before we went our separate ways. Apparently, they liked my headdress. Lovely folks.

I carried on walking, in my blissful state, up past the Left Field stage, where I stopped briefly to cut short some little kids who were trying to take the piss. I blind-sided them with good temperament and wished them a good festival. I still don’t quite know why, it just seemed easier than ignoring them.

Before long I was up by the Acoustic Stage, sat outside it and returning the ever-increasing number of “Hows” I was receiving, even occasionally going as far as responding with “That’s How… for now [dodgy facial expression]”. It amused me greatly.
Up outside the acoustic stage I got talking to Tom and his Czech girlfriend (Katka?). Lovely people. And really quite interesting. God only knows what they thought of me. All I really remember is repeatedly apologising for ruining their evening. I sat outside the tent talking to them most of the way through Sinead’s set. Her set provided great background music.I also spoke to a group from New Zealand and did a crap job of taking their photo for them numerous times. Even the time I tried to rest the camera on Tom’s head didn’t work. Wacky.
We sat, chatted, had a little smoke to stir things up and laughed about Tom’s apparent knee fetish. Fun times.

After staying just long enough to thoroughly embarrass myself I left to go catch the Kings of Leon set. Feeling fairly content still, the walk there passed quickly, with a silly grin slapped across my face. More “Hows” were exchanged.

Wandering around the crowds waiting for the Kings of Leon, my initial intention was to find Mike and John, who I knew were both meant to be there somewhere. Apparently, they were equally smashed after getting through as much of the booze as they could. Solid performance.

However, I didn’t find them, as I was distracted by a fun-loving group who stopped to talk to me. After providing them with my carefully rehearsed caution – “I warn you, I am an idiot and have spent the evening attaching myself to different people as I can’t find my friends” – we joked, laughed about my attire, and smoked some more, and waited for the show. It was fun, but I sort of drifted away from them mid-Kings of Leon as I seemed to be struck by a niggling sense of paranoia / soberness that left me realizing I may be ruining everyone’s night _and_ the fact that they seemed to think I was gay. I questioned what I was doing, and just switched off and watched the rest of the set on my own. That was rude of me, and I apologise. As with most people I met that night, they were a lovely group, and I shouldn’t have let my social ineptitude stand in the way of me trying to be polite. God damn my shy side!

Before the end of the set, I was talking to another two strangers briefly, as one of them wanted his picture taken with the headdress on. I said he could, provided I was in the picture. He agreed, the picture was shocking, and I apologise for ruining it. Besides that, I don’t remember much of the conversation.

As soon as that set finished, I left and set off for The Park, to see the end of Pete Doherty’s set. I’d forgotten how long a walk it was, and arrived just in time to catch the end of it, although I very nearly didn’t see anything – the multicoloured tower in the corner caught my eye and I walked aimlessly towards it… and past the stage.
You can’t make this shit up.
So, after doubling back on myself to see the end of Doherty’s set, I was at a loss as to what to do I realized I was still on my tod, 5 hours after I’d last seen any of my friends, and I was sobering up.

I took a stab in the dark and guessed that the rest of them might be in the Dance Village, so I set off down there feeling distinctly alone.
When I couldn’t spot them, I turned around and walked back up to the tent (near The Park). When they weren’t there, I went _back_ down to the Dance Village where I got a little bit worried when one reveller threw his arms round me and tried to tell me that I had his hat. I don’t remember what exactly I said to him, but I think I took his comments as more threatening than they were intended. So, sorry for that.
If they were meant threateningly however then you, sir, are a cunt, possibly the only one I met that weekend.

As I got back to the tent, John and Mike arrived, both thoroughly drunk. I was glad to see them and chirped up a lot. Shortly after John threw up and passed out. I took photos. Mike had a Pot Noodle (I think). Then we grabbed some Strongbow and headed up to the Silent Disco in The Park, for all of about 30 minutes, after which it closed, and we returned to camp to sit around and talk. Mike slept, and I decided to wait up for Owen and Duncan, like a paranoid parent. I still don’t know why, I guess I just wasn’t sleepy.

And that kids, is the story of Crazy Friday. Or, at least, my interpretation of it. I’m sure the others have their own specific stories and fables as to how their evening went, but that was mine. It was truly reckless, spur of the moment, and surprised me in that regard, but it was also just pure fun, and the first time I’ve ever really felt an significant effect from pills. Very interesting, if not very clever.

The Food

Not the cheapest but – when you balance it off against the circumstances – not ridiculous either. I believe Mike estimated, and we concurred, that you could easily buy every meal of the day for £15 each day. Which really isn’t as bad as it could have been, as that estimate is on the generous side in as much as we would feel full after such a day
We, however, took many Pot Noodles with us, and a good few cans of beans (some even with added sausage!) and a spot of soup. We ate (relatively) well, but I don’t want to look at another Pot Noodle for at least another year, no matter how good they were at the time..!

Of course, it also fails to factor in beer. I’m glad we took our own.

Averaged £3 a pint, which for a festival isn’t bad, although there were surprisingly few bars offering the bitter.

However, the Hot Spiced Cider stall went down a treat – total win. And, dare I say it, fecking lovely.


Surprisingly few, really.

One big one is that I didn’t get to see Duffy. I really should have forced myself off my arse and gone and seen her. Nothing wrong with the Raconteurs, but I do rather like the Welsh girl.

Another one would probably be for the few artists I really wanted to see but didn’t manage it – namely Groove Armada, the full Massive Attack set, and Fatboy Slim.

Yet another one would be that I didn’t spend more time around The Park. Some good stuff on up there.

One more? Missing out on Tony Benn because of a timetable change. I actually wasn’t that bothered to see him pre-Festival,  but the disappointment of a later schedule-change bothered me. Fucker. 😉

Final one? Not taking the Monday off work. Not only because it nearly killed me, but because I would have rather enjoyed going fookin’ crazy on the last night. That would probably have killed me as well.


Glastonbury really doesn’t need a conclusion. It met and surpassed all my expectations.

However, one caveat – keep the variation. Maybe hip-hop doesn’t warrant a headline slot yet, but it needs a place. Good hip-hop is, well, good. And it was a brave choice. So bravo.

If you get the chance, go to 2009. And if you do, let me know. We’ll beer.

‘Intro to Caving’

Monday saw me taken on a little excursion, to Iron Kiln Hole (Notts II) up at Leck Fell (info at http://www.rrcpc.org.uk/easegill/text/leck.htm ), with Rob Dean, Danny Fitton, and Dez (surname unknown, to me at least), basically to see what I thought about caves and what all their fuss was about… 😉

It was the first time I’d ever really been caving, be it scuba diving or, as in the case, (relatively) dry. I didn’t take a camera, which was both a mistake and a saving grace (I would have managed to trash it), so unfortunately what follows is just my experiences of it.

I met Rob, Dez, and Danny in the Cafe above Inglesport in Ingleton just after 2 in the afternoon which, to me at least, seemed an odd time to be just considering setting about doing something, but then I guess that’s from a diving mindset of getting places early and spending the day doing it, given the time constraints involved in being underwater.
They’d all just about finished their food and brews and so we headed more or less next door to hire me an oversuit, undersuit, and helmet. I brought my own wellies… 🙂 Then we set off in convoy, following Danny up to Leck Fell to leave the cars, get changed, and head down to the entrance to the cave – effectively a shockingly small hole covered with a man-hole cover which, for some reason, was not the entrance I was expecting to find. I say ‘for some reason’ because, whilst I expected caving round these parts to be small and tight, I still can’t explain why but I wasn’t expecting a vertical shaft access to it. Just call me naïve.

I headed into the shaft third, with Danny closing the way and replacing the cover. the LED lamps on the front of the helmets actually provided an impressive amount of light across quite a wide beam, which was nice. Straight from the outset it was about climbing down scaffolding and a bit of breeze-block-fashioned ladders, which was interesting, although admittedly I was stuck wondering at first quite how long this would go on for. Judging by how quick we seemed to get back up it on the way back, it really wasn’t as far as it seemed.
To me, the bit at the bottom was where it got interesting, if a little difficult. Getting pointers from Rob in front and Danny behind about how best to twist and turn my body to get over or around various obstacles and tight gaps was pretty entertaining at times, as I never quite realized how much movement you can have when you really need / want to. It was also pretty refreshing to be having to think about things in a similar sort of way as I had to when I was out climbing with Dicko and Rob.

At the bottom of that little stretch it all started to level out a bit and opened up into a much larger chamber than I was ever expecting to find. This was only really the start though, and the next part was a damn sight tighter and closer than I had planned on as we scrambled up a mud slope into an insanely low section, crawling on bellies for what seemed a bloody good distance until it opened up enough to, initially, sit up and take a breather and, eventually to stand up and walk around a bit, as well as seeing my first sight of some nice stalactites, by which I mean ones that weren’t around 6cm long. As ever, it sounds stupid, but I really wasn’t expecting to see much in the way of stalctites and the like as I simply wasn’t expecting to be in passages big enough to house ‘big ones’. As ever, I couldn’t be more wrong.
Back into another tight scramble we went, at which point I was getting a little irritated, not at the scrambling, but at the though of “if this doesn’t go anywhere, there’s no way I am going to be able to crawl backwards out of this bastard. And I sure as hell can’t turn around”. It wasn’t really the tight scrambling that annoyed me, in its own way it was actually quite fun, it was the fact that I was failing to get used to the idea that I couldn’t look straight ahead, as my helmet would keep hitting the roof, so I had to look to the side and angle my body accordingly.
Anyway, this didn’t go on too long and, to be fair, it was all worth while as we came out the other side above a fairly fast-flowing stream that had clearly carved it’s path through the surrounding rock over a number of years – exactly the sort of thing that, for whatever reason excites me. Maybe it’s just the image of running water that I’ve always sort of found fascinating.

We headed further up against the stream through the fairly narrow but high parts and carried on that way until the flow slowed off as things got a bit wider, flatter and, in parts, deeper. We carried on up that way, towards the sump (I can’t pretend I remember where it’s meant to lead) and Rob and co. were commenting on how the water level was a lot lower than the last time they were there. That struck me as odd as it started to reach testicular level on me… shortly after that though we were, indeed, at the sump, and had to climb up and across to take another small-ish route which involved a lower rough, making crawling on hands and knees the ideal approach (if a little uncomfortable without gloves and, before rob gave me his elbow pads, any knee protection).

At this point Rob and Dez continued a pretty intense mud fight, which was entertaining to spectate  on at least. The next stretch continued the mud theme, along with the crawling, all of which was easy enough, although I was becoming increasingly more aware of the added weight from having a suit that had become fairly waterlogged (if not actually too cold) and still thought we were going to have to crawl on hands and knees for a large distance on the return journey. After passing some more impressive formations, myself and Rob stopped whilst letting Danny and Dez go ahead and, quite literally, test the water at the next block they came across. To say it was deep would be an understatement. Turns out it was another sump, apparently leading to ‘a huge chamber’ (Danny’s words, not mine). But to get there we were going to have to swim across it.
I think if we’re honest, Rob and I were all for turning back, but we cracked when Dez came back to encourage us. I have to admit, even when I was stood waist deep in the water on the edge of where I was going to have to swim, I was having doubts, but I figured I’d got that far, I might as well give it a whirl. Besides, the rest of them were already on the other side. Turning back would have made their swims a waste of energy. So I finally went for it.
To say it was cold would be an understatement. It was fucking freezing. The sort of cold that takes your breath away (although, admittedly, I was impressed with how well the simple wooley bear and oversuit worked at slowing the influx of water).

So we’d manned up and swum across the gap (probably a couple of metres across). We then got out, shook around a bit to try to keep warm, and set off towards the ‘chamber’ Danny had promised. It got narrow, and before long the water got deeper again. Expletives were uttered, albeit in a light-hearted manner, and we turned back, for another swim. It was no warmer, but we were having a laugh at least.

The way back was largely uneventful, following a similar route. Before we got back to the spot we’d belly-crawled through, Danny took me on a diversion to show me Curry Inlet. There were some simply stunning formations on the way to it and I’d love to go back and spend a little more time around the area just admiring the formations.
The route back took a slight variation as we got to the point where we’d crawled through as, instead, we followed the stream down and winding around through the narrow passage back to where Danny and Dez had left the bags. This was a great way to end the walking around as well, for me at least. All down hill, water rushing all around, various rocks that had fallen lodged at different angles to duck and weave around. Great stuff.

As mentioned right at the start, the climb out was both a lot easier and a lot quicker than it had felt on the way in and before long we were back above the surface, where the visibility had improved, providing a simply stunning view right out down the valley and over the bay towards the offshore wind farms. Again, the lack of camera irritated me although, to be fair, if I had one it probably wouldn’t quite have captured what a good view it was.

Time to change again, try and put dry clothes on, and then down to the Marton Arms for a swift point before the drive home.

Overall impressions? It was a lot of fun, and in many ways lived up to everything I expected it to be. I wouldn’t say any of it particularly surprised me, although I did expect to find generally a lot more smaller passages and a lot less of the open spaces that were there, along with the formations.
I don’t know a lot about caves in general, and in a way I kind of like that, as it leaves me just appreciating things as they are and as I see them, rather than knowing all the reasons why they’re like that. Of course, it would be nice to know why as well, but I’m happy to have Dez, Rob and Danny tell me instead for now!

Would I go again? Definitely. I mean, it’s not something I can see myself ever really getting into and being as passionate about as I am with my divin but, just like with climbing (which I really need to start doing more of) I could definitely se myself doing it every so often, having a laugh and just getting out and about.

And that just about wraps it up. I’m pretty sure there was more I initially wanted to conclude with, but I guess if that reappears in my head I’ll add it later. Muchos thanks need to go to Danny, Dez, and Rob for letting a numpty such as myself tag along with them and for showing and teaching me some really cool things. Was definitely a good way to spend 4 hours in the dark!