Home NAS & Server (5): Tweaks, Quick Tests, and Data Transfers

This is Part 5 of a series of blog posts on building a Home NAS & Server using Linux. For the full index and other details, please see the Index

Once things are set and up and running, the first step really is to make all those tweaks and short tests that we want to do to ensure things are running as we want, and then migrating our existing data over to the new system.

Whilst prepping this particular post, I went back to try and find my references – particularly for the tweaks. As Murphy may well have predicted, I couldn’t find them. I’ll do my best on each point to either find them after the fact, or at least state where I saw them and explain them as best I remember, but I apologise for any I’ve missed or miscredited.

Limiting ARC

There’s a bunch of decent references for this on Solaris Internals ([1] and [2]) and some good threads covering how to do it on the zfs-discuss list.

There are a few reasons you might want to do this but it’s worth noting that the defaults in ZFS on Linux were changed from those Solaris used so that it defaults to 50% system memory – https://groups.google.com/a/zfsonlinux.org/forum/?fromgroups=#!searchin/zfs-discuss/arc/zfs-discuss/4iNT8aXouzg/UL6xf69HkjMJ

Additionally, this thread explains how you might make such changes, and introduces a couple of other options you may consider, particularly if you’re planning on running zvols (for block devices) and so on.

At this point I’m not making any of these ARC changes on either build until I’ve monitored how things pan out performance-wise, but they’re good to know about.

Disabling / Limiting Swap

Reference: https://groups.google.com/a/zfsonlinux.org/d/msg/zfs-discuss/PzBAABrZCb4/5Z-tscgBBaQJ amongst other places.

Anyway, the basic premise here is that “normal” swap is your enemy – and your weak link in the chain – in terms of data protection and checksumming in  a ZFS on Linux capacity.
Why? Well, the basis of it is down to the difficulties in easily running ZFS root under Linux. I haven’t tried to do this yet, but the difficulty comes in being much more careful on how you upgrade things where the root is concerned – especially if you want to keep the experience pain-free. Because of this, a lot of people will install root on “normal” drives, including swap. Once that’s done, if you ever use that swap space (to help avoid out-of-memory conditions) you’re sacrificing all the hardwork and additional money you put into ECC RAM – your swap has become a single point of failure.

You can, of course, create a small ZFS filesystem to use as your swap partition. Or have a large amount of RAM that (like in my position) you may not touch for a long time, but an easier method may be to just turn “swappiness” down to 0. I’ll let you reference the various pages about this yourselves but, effectively, this will all but disable swap unless the system really, really needs to use it (to avoid out-of-memory). The easiest way I found to do this (and make it permanent) was to add a file (swappiness.conf) in /etc/sysctl.d/:

~$ cat /etc/sysctl.d/swappiness.conf
# Only swap to avoid out-of-memory
vm.swappiness = 0

This will get enacted on the next reboot, or you can run sysctl -p /etc/sysctl.d/swappiness.conf to force a read of the options in the config file instantly. Similarly you can echo the “0” straight into /proc. The important thing is to make it persistent across reboots.

Copying Data from the QNAP

Going into this, I barely thought about the data migration. It seemed simple – rsync, right?

It made sense. The QNAP can act as a rsync server with ease (Administration Panel > Applications > Backup Server) and, once done, there was no problem installing rsync on the new NAS and connecting to it with:

# rsync -avz admin@nas:/share/Public/ISO/ /srv/multimedia/ISOs

A quick prompt for my QNAP admin password and it starts syncing, compressing the stream and maintaining attributes.

There was one major issue though – it took bloody ages. Seriously, a long time. Almost a day. And that was just for the music directory, which is only around 110GB in size (Films alone occupy more than 1TB). The average stats rsync reported at the end of the transfer happily told me it had averaged a mighty 3MB/second. Put simply, that was not the performance I was looking for.

I freaked out a bit. Raidz2 couldn’t really hit performance that badly, could it? None of the research I’d read suggested it would. I ran a dd to one of the newly created filesystems, just to check the disks were behaving alright. Performance was good. I didn’t write down the figure, but it was around 800MB/s – the disk layout isn’t the issue.
It shouldn’t be compression (lz4 is enabled by default across the pool) – all the supporting documentation suggests the overheads are minimal and, with modern hardware, there’s little reason not to enable it pool-wide, even if the gains are minimal for some pools.
Is it the network? I’ve got a fairly respectable Zyxel GS-2200 Gigabit switch that, if anything, is massively under-utilized. Besides, nothing else but my desktop was using the network at the time. It seemed unlikely that was the issue either.
I tested without the “-z” (compression) – no joy. I tested a transfer of some extra ISOs from my Windows desktop over SMB – I got a respectable 100MB/s out of it, even with the SMB overheads. Strange.

I researched a little bit and came across a couple of posts that pointed to SSH and the encryption overheads as being likely to slow things down. I doubted they were doing much to tax the new system but it seemed reasonable that they would potentially kill the TS-219P, which – as we covered earlier – is not exactly the meatiest of machines.

As my TV Shows transfer was still running (over the default method, SSH) I figured I’d try the points raised in those posts and mount the NAS locally over NFS and then rsync “locally” from the mountpoint to the ZFS volumes. Comparatively speaking, it flew out the door. Here’s a couple of quick rsync summaries:

rsync TV-Shows folder (SSH):
sent 55932 bytes received 674416624286 bytes 5175538.67 bytes/sec
total size is 674334091593 speedup is 1.00

rsync Live-Shows folder (NFS):
sent 18278740591 bytes received 660 bytes 54644966.37 bytes/sec
total size is 18276506379 speedup is 1.00

It may not be the most scientific of tests, but it’s around 10 times faster. Off the back of that I’m now confident enough to try and move the Films all in one go, with “-z” added back into the mix. Hopefully, it won’t still be running this time next year.

In transferring everything with rsync, I noticed another little issue I hadn’t noticed / been aware of before – a massive amount of “.@__thumb” directories. A quick search verified their source – the TwonkyMedia server running on the QNAP.

I don’t intend to use Twonky for DLNA on the new build, nor do I have any desire to keep a bunch of useless hidden folders lying around my filesystem. I knew I could get rid of them using find, but wasn’t entirely confident on my syntax. This page helped, and in the end I was able to do it comfortably with:

find /srv/multimedia/ -name ".@__thumb" -exec rm -r {} \;

It had soon whipped through even the more verbose directories (like the Music folder) and removed all the junk. As far as I can tell, no genuine files were harmed in the running of that command – nor can I find any reason why they should have been.

With the exception of the films currently transferring, the data migration has been painless and straight-forward. The slow speeds were obviously a concern but at least the result of that was application- / protocol- based rather than something more fundamental. Right now though, the basic data is in place and sitting pretty:

:~$ sudo zfs list
pool1 946G 4.26T 232K /pools/pool1
pool1/backup 28.8M 4.26T 198K /pools/pool1/backup
pool1/backup/backuppc 28.6M 4.26T 28.6M /var/lib/backuppc
pool1/home 239M 4.26T 198K /pools/pool1/home
pool1/home/dave 239M 4.26T 239M /srv/home/dave
pool1/multimedia 946G 4.26T 198K /pools/pool1/multimedia
pool1/multimedia/Audio 107G 4.26T 107G /srv/multimedia/Audio
pool1/multimedia/Books 267K 4.26T 267K /srv/multimedia/Books
pool1/multimedia/ISOs 2.54G 4.26T 2.54G /srv/multimedia/ISOs
pool1/multimedia/Photos 2.59G 4.26T 2.59G /srv/multimedia/Photos
pool1/multimedia/Video 834G 4.26T 834G /srv/multimedia/Video