Home NAS & Server (4): Setting up ZFS

This is Part 4 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index


EDIT – April 2013: Since originally writing this segment, ZFS on Linux has reached it’s first non-RC version – 0.6.1. With that release, a Debian Wheezy repositories are available that handle the install process. If upgrading from these instructions, the previous packages must be purged fully before installing from the repo (dpkg –purge zfs zfs-devel zfs-dracut zfs-modules zfs-modules-devel zfs-test spl spl-modules spl-modules-devel). Up-to-date Debian-specific instructions are at http://zfsonlinux.org/debian.html)

Fortunately, the good folks involved with ZFS on Linux have got both useful instructions on compiling your own packages along with a lovely Ubuntu PPA repository for both stable and daily releases. Conveniently the Lucid PPA works perfectly with Debian Squeeze, although I’ve settled on Wheezy for the HP ProLiant Microserver, so for that I’m building from source.

Compiling and installing ZFS

In testing (within a virtual environment) I followed the deb building advice for my Squeeze machine, and found the only thing I needed to do was run update-rc.d zfs defaults after install to ensure that the pool is automounted on boot. They were just as painless for Wheezy, and needed no trickery to automount the zpool on boot.
Using the PPA on Ubuntu I had no such concerns.
The PPA pages detail all that is really needed to add the repository to Ubuntu, and installation is as simple as aptitude install ubuntu-zfs but there were a couple of different steps needed for the Debian Squeeze system. As Wheezy shouldn’t be too far off become the new stable, I won’t spend any time on those steps – Google is your friend.

Once it’s finished installing and the kernel modules are compiled you should be able to run zfs and zpool commands.

Preparing zpools

I won’t be using hot swap / external loading drives on either of these builds, at least not to begin with, so having a relatively painless way to identify which disk is which is relatively important to me. To that end, I decided to use the vdev_id.conf file to allow me to assign human-readable names to all the installed disks. As detailed on the project page, this technique is really more useful in much larger environments to ensure you can quickly and easily identify separate controllers for greater redundancy in your pools. In my case it’s more so I can quickly and easily identify which disk is broken or misbehaving when that time comes. A quick cross-reference of the disk serial numbers before I inserted them into the bays with the device assignments once booted helped me confirm the correct PCI addresses. The naming I decided on was:

~ cat /etc/zfs/vdev_id.conf
#
# Custom by-path mapping for large JBOD configurations
# Bay numbering is from left-to-right (internal bays)
#
#<ID> <by-path name>
alias Bay-0 pci-0000:00:11.0-scsi-0:0:0:0
alias Bay-1 pci-0000:00:11.0-scsi-1:0:0:0
alias Bay-2 pci-0000:00:11.0-scsi-2:0:0:0
alias Bay-3 pci-0000:00:11.0-scsi-3:0:0:0

After doing the above, running udevadm trigger will cause the file to be read and the symlinks to the correct devices will appear in /dev/disk/by-vdev/. The benefits should become clearer once the pool has been created.

For reference, Bay-0 is my root drive – I’ve not opted for ZFS on root, nor any real resilience due to space constraints (I may work around this in the future). For the rackmount build – as mentioned – I’ll be looking at small 2.5″ drives mirrored using mdadm for the root files.

At this point, all that’s left to do is create the pool. As mentioned, Bay-0 is my root disk, so the disks I’ll be looking at using are Bay-1Bay-2, and Bay-3 – configured as raidz. To confirm things, I ran zpool create with the -n flag to verify what would be done. Once happy, the -n can be removed and the command run:

zpool create -f -o ashift=12 -O atime=off -m /pools/tank tank raidz Bay-1 Bay-2 Bay-3

A couple of notes on this:

  • I used -f because I received the following error without it:
    invalid vdev specification
    use '-f' to override the following errors:
    /dev/disk/zpool/Bay-1 does not contain an EFI label but it may contain partition
    information in the MBR.

    I’ve seen this error in my testing as well, and found a few threads on it, but nothing convincing. I confirmed with fdisk that the disks do not, in fact, contain any partition tables, and opted for -f to force past it.
  • ashift=12 is per recommendation (drives with 4k blocksizes, per http://zfsonlinux.org/faq.html#HowDoesZFSonLinuxHandlesAdvacedFormatDrives
  • atime=off as a default setting for the pool filesystems as I can’t see any real reason to enable it across them all.
  • -m allows me to specify a mountpoint – whilst I’ll only have one pool, I figured specifying a specific pools directory would be a good habit to form if I ever want to import other pools in the future. The directory must exist before this command is run.

Once created, things can be confirmed with zpool status:

~ zpool status
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
Bay-1 ONLINE 0 0 0
Bay-2 ONLINE 0 0 0
Bay-3 ONLINE 0 0 0

At this point I’m ready to do pretty much what I want and can verify my pool is online and mounted with df.

Home NAS & Server (3): Hardware Choices

This is Part 3 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index


Once the software choices had been settled on, it’s easier to get a clearer picture on what I want to get from the hardware along with quite what spec I need.

Case

Having initially wanted to make the new case fit into a 3U max space (which is what I happen to have free in my cabinet at the moment) I was getting a bit nervous when trying to find options. What I began to find was that the vast majority of the available options will easily fit in 2U or 3U, but most end up requiring 26″ depth, which I don’t have. The smaller cases were largely ruled out as they required boards that couldn’t take the amount of RAM I’d want, or didn’t have the requisite disk / expansion slots to cater for the storage side of things.

For a long time I was looking at the Antec rackmount cases (2U/3UEPS systems) but found them hard to source and I was concerned about getting tied in to specific (expensive to replace) PSUs. As time’s gone on it seems they’re hard to find because they’re being wound back, if my recent visit to the Antec website is anything to go on. I later shifted to looking at the Chenbro RM-22300 because it’s seriously cheap, and meets the form-factor requirements. As an additional bonus it also fits standard PSUs, although it looks a bit of a cludge in how its cabled. It’s big drawback is that whilst it would fit the bill today, it lacks room for any real expansion.

Looking back at things more recently, I’ve figured I can safely sacrifice my 2U shelf in order to make room for a 4U case. Whilst I don’t need anything that size right now, it pretty much allows for a full-sized desktop case which means normal PSU, lots of drive bays, and plenty of easy-to-facilitate expansion. Provided I get some fixed rails to support the weight, there’s no real reason not to look at it.
As I liked the look (and reputation) of them already, I went back to the Antec 4U22EPS650 option which comes with a PSU and has plenty of room and options. One nice point is it can potentially be expanded to have 3×5.25″ slots available on both sides of the rack, allowing for hot-swap bays to be added fairly painlessly. It’s easier to find than the other options and will easily address any future expansion concerns.

Hot Swap Drive Bays

Whilst not exactly a key requirement to be in place on the new build from the get-go, easily accessible drive bays is something I’d definitely like to have in place.

I’ve been looking at a few possibilities, but the one that still leads at the minute is on of the 4.35″ in 3×5.25″ options from IcyDock. Not being in a position to get one yet, I haven’t quite decided whether I want to go fully tool-less or not, but price will probably be the main deciding factor. Otherwise, they’ll all do SATA3 which is my only real concern.

Processor

As I’ve already covered, one thing I am keen for this box to do is cater for other projects beyond the scope of a “standard” NAS, including acting as a hypervisor and a PVR backend. So it could do with a bit of oomph in the CPU department, whilst at the same time being conscious of the fact it will be on most (if not all) hours of the day. Given that I’ll be using ZFS for my storage, it also needs to be (ideally) able to work with ECC memory. The more I looked at available desktop options, the more I came to realize that using one of the newer i3 / i5 / i7 Intel chips would mean sacrificing ECC or paying a lot more for enterprise boards – which I can’t justify at the minute. ASUS, on the other hand, offer consumer boards that support ECC merrily in their AMD line.
So I got looking to AMD CPUs. Looking at the stats, the energy efficient ones have considerable savings, but were hard to find in the UK. Not impossible though, and I managed to pick up a 615e on import. I just needed a Heatsink / Fan for it, which eBuyer helped with.

Motherboard

Having already come down on the side of an AMD CPU and ECC RAM, I was looking fairly firmly at Asus to help fulfil the motherboard requirement, and I don’t think they’ve disappointed. The M5A99X EVO ticks all the right boxes and caters for a good number of hard drives (6 using SATA 6Gb/s ports and 2 using 3Gb/s) along with ECC RAM support.
As with the CPU decision, power usage here won’t be as low as I might have wanted from a “pure NAS” box, but it will should have more than enough grunt for the additional tasks I want it to perform.

Memory

The MB can take up to 32GB ECC RAM, but I found 8GB ECC Unbuffered DIMMs hard to come by. All the main manufacturers’ sites have them, but finding them at suppliers was more of a challenge.

In the meantime then, I’ve opted to settle on 4x4GB DDR3 PC3-10600DIMMs from Kingston, KVR1333D3E9S/4G. The pricepoint is pretty good for my plans, and 16GB should give me plenty to work with.

Longer term, 32GB is the plan, using 4 8GB DIMMs, but I can expand to that when needed.

Hard Drives

At the same time as I first started thinking about doing this build, Western Digitial released their Red drives, aimed specifically at the SOHO NAS market. I’ll be honest, I don’t know a vast amount about the intricacies of hard drive technology – and I do appreciate that “Enterprise” / “Server” drives will always be more substantially tested than SOHO / Desktop equivalents – but in the reading I did around these, the implication seems to be that they’re worth a go. Firmware improvements, longer-than-normal warranty and being pitched (and tested) as always-on seems to make them a reasonable choice. I guess time will tell.

When I first set about this, I sort of default assumed I’d be looking at the 2TB drives as a price compromise between safety and resilience, either in a mirror or raidz (RAID-5, effectively) configuration – so tolerable of 1 disk failure. However, recent discussions haveencouraged me to look towards using the 3TB disks, but upping to raidz2 (double parity).

For the boot disk, I’ve opted to go with a single 2.5″ SATA2 drive from WD. I might eventually up this to a software mirrored RAID option, but as all the important details will be backed up to the ZPool anyway, downtime for a single disk failure isn’t the end of the world (at this point).


Side Note: The Other Build

As noted in passing earlier, I am also building a similar (but slightly lower spec / requirements) machine for my parents. The intention is this will serve both as a central store and backup machine along with serving as an off-site backup for myself using snapshot send / receive.
This machine wouldn’t need to do the extra stuff I want the rackmount box to do. It doesn’t need to PVR and it doesn’t really need to host VMs. It needs to hold data and run some relatively light services for the rest of the network.

Given the £100 cashback offer that HP are still running, I opted for a HP ProLiant N40L Microserver, replacing the 2GB RAM with 2 x 4GB ECC RAM modules and adding a StarTech PCI-E Dual-Port Network Card to add the additional ports the box will ultimately need. For disks, I used three Western Digital Reds, 2TB each, and kept the 250Gb that came installed in the system as a root drive.

Home NAS & Server (2): Software Choices

This is Part 2 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index


Given the scope of what I want the new build to do, it feels wise to decide what software I want to use to provide those capabilities, as it’s more than likely that some of those choices will effect the hardware specifics they need to run upon.

Looking back over the previous list I feel there’s a few key points to cover, namely Operating SystemFilesystem / RAID, Virtualization, and DVR. There’s probably more besides that I’ll think of later.

Operating System

It’ll be no surprise that the OS is going to be a flavour of GNU/Linux here (it’s sort of the point of the project), but I should probably justify both why, and which flavour I’ll be looking at using.
Using Linux is the obvious choice to my mind based on a number of factors, not least of which is familiarity (as a server platform) combined with my general preference for Open Source software. It’s cheaper too, of course, although to be honest that’s less of a major factor given that Home Server versions of Windows aren’t massively expensive these days. The thing is, I’ve never really liked Windows Server stuff, even now I’m in a position to have access to them through labs at work, they just feel clunky. Solaris / OpenSolaris / Illumos, too, I disregarded as I’m not as familiar with it and it, frankly, doesn’t feel as flexible as my experiences with Linux.
I considered BSD-based solutions which, whilst viable, I’m less sure I can get things like the PVR-backend functionality (and TV Tuners) working on it without considerable (additional) messing around. I’m going to try and get more familiar with BSD, but I can’t justify this build as being the time to do it – given the Swiss Army Knife approach.

In terms of the distribution, I originally came down on the side of Ubuntu 12.04 LTS Server Edition, with a view to moving to Debian 7.0 (“Wheezy”) when it’s released. However, with Wheezy’s release being so close, I figured it’s reasonable to start with it straight out of the blocks.
Debian’s what I’ve done most of my playing around with and I’m a big fan of APT for package management. However, I’m less confident of getting TV Tuners working with Squeeze, unless I start to draw more heavily on Backports – if I’m going to do that, I’d rather use Ubuntu LTS in the interim, and jump to Wheezy when it’s “new”.

Filesystem / RAID

When I set off down this path of building my own NAS I genuinely hadn’t given much thought to how I was going to handle this aspect. In my head, it was obvious – mdadm software RAID (I can’t really force any justification for using hardware RAID), most likely with LVM on top of it and most probably ext4 as the filesystem.

The more I got to thinking about it though, the more I got comparing some of my desired features with the kit I get to play with at work (NetApp filers of varying size and shape). Whilst I had previous exposure to snapshots through ZFS on Solaris, I never fully appreciated quite how powerful and useful they are. I like the idea of being able to do relatively easy, incremental backups to a remote location; I like the scalability of it (even if the scale is way beyond anything I’d need for a home NAS); and I like the extra steps it takes in checksumming and data integrity. Given the way ZFS handles disks, it effectively takes care of the software RAID side of things at the same time.

The downside is that ZFS isn’t really native on Linux (I’ll let Google help you, but fundamentally it’s a licensing issue). There’s a ZFS-FUSE project but as you might expect, all the reports suggest performance is sluggish to say the least. Additionally, I’ve already ruled out using Solaris / BSD as a base. There is, however, a native port maintained as ZFS on Linux which has a pretty good implementation working (albeit not quite with all the features of the very latest ZFS). I’ve been following the project for a while and there’s some healthy activity and, generally, I’ve been impressed.

To keep things native, I also looked at BTRFS which seems to be showing a massive amount of promise but, as of yet, doesn’t quite tick all the boxes to me on older / non-bleeding edge kernels. It’s something I definitely want to keep an eye on and test further though, as it seems to have the potential to surpass ZFS as time goes by, especially with the continued uncertainty after Oracle’s takeover of Sun and whatnot.

So, whilst it flies a little in the face of some recommendations from friends, I’m deciding to trust a collection of things I read on the internet and going with ZFS On Linux at this stage.
The key point for me was send / receive snapshots (on a relatively stable kernel version) – as I’m planning to build a similar device at my parent’s place, it painlessly addresses my off-site backup desire.

Virtualization

Given the way I seem to be wired, when I first considered having this build server as a platform for a few VMs (both for “production” use and for testing environments) I got way ahead of myself in considering whether running the host as dom0 for a Xen hypervisor, or using KVM. In both cases, I was heading down a road of working out whether running the hypervisor kernel would have knock-on effects on the systems desired output (NAS / PVR) and how to mitigate that.

Eventually, I realized I was being ridiculous. At most, I’m going to be running between 3 and 5 VMs near-constantly, with maybe a couple of extras fired up for lab purposes. None of them are going to be particularly resource-intensive, nor should they be under high load. As much as anything, they exist to provide some sandboxing and allow me to mess with config management and other tricks and tips that can ultimately be useful and applicable in a work environment.
Before anyway chooses to state the obvious – I can appreciate that Xen / KVM would be pretty good skills to have, but shoe-horning them into my domestic NAS environment strikes me as overkill (and if it *strikes* me as overkill, it *definitely* is overkill).

In the end I think I’ve setted on using VirtualBox, at least in the interim, for a few reasons. I’ve got some experience with it already in headless mode (albeit, not enough); it provides enough of the passthrough features that make it versatile-enough for my needs (I run my current labs in VirtualBox on my desktop); decent management tools exist; it can do some fun stuff to aid the integration with the neater features of ZFS; from what I can gather, the key limitations really only manifest themselves at a business-level, rather than home-use.

DVR

As has been hinted at previously, I really have no firm alternatives to MythTV for this purpose.
I first saw MythTV in action first-hand at a LUGRadio meetup many moons ago and I was wowed. Whilst I don’t doubt that there are other alternatives available (most, to my knowledge, need Windaz) – and certainly others that are quicker to configure – the sheer scope of what MythTV has achieved appeals to me massively. And I want to be – in some way – involved.

But, I’ll be honest, DVR had taken a bit of a backseat for me in short-term plans due to the difficult integration with XBMC but then these two posts ([1] and [2]) got me excited about it again. The only difficult part will be choosing a decent, supported tuner.