Setting up the Sheevaplug as a Weather Station

UPDATE 25/5/2011:

$ uptime
07:42:35 up 94 days, 14:00,  1 user,  load average: 0.74, 1.32, 0.68

Pretty happy with that so far!


A few months back I got hold of a Marvell Sheevaplug from NewIT with the express intention of it replacing the aged, noisy desktop PC with a more efficient, lower-power and quieter solution. It’s sole intention was to run the open source weather station solution Wview – a tidy little collection of daemons that records the weather statistics provided by a variety of different weather station hardware and then allows you to do a bunch of things to it (generate your own HTML templates, send the data to Wunderground, Weather for You, and so on) .
With that in mind, my key spec and concerns for the device were as follows:

  • Run Debian Squeeze
  • Fetch data from an Oregon Scientific WMR968 via a USB-to-Serial adapter
  • Record the data consistently using WView
  • Given the risk of the SD Card failing, regularly dump a backup of the data to a remote store
  • Take steps to minimise the effect of regular writes to the SD guard and so prolong its life

When I initially got the SheevaPlug from NewIT, it already did run Debian Lenny, and I started taking steps to stick things like Flashybrid on there. However, I was hasty, and it ended up breaking the install.
Given that I was going to have to do it all again, I thought I’d best document it, especially as I could only find the information I used spread out over 4 different websites (see References, bottom). As such, these details are pretty specific to the exact steps I took, rather than the more general (and detailed) information that can be found at those references.

PLEASE NOTE: My installation environment was onto a standard SheevaPlug (no ESATA port), installing from a TFTP server onto the SD Card, and making that SD Card bootable. If you’re planned install environment differs to that, use Martin’s site to see what you need to do differently.
For all codes and commands, I have used
#~ to indicate the commands are being input on a new line.

Initial Setup and Installation

All credit for the initial installation and setup has to be given to Martin Michlmayr at cyrius.com[1]. I more or less followed his instructions to the letter, installing as I was to an SD Card and using a TFTP Server running on my laptop to facilitate the install of the uImage and uInitrd. I would also recommend that your source your images direct from his website, I won’t be reposting them here.

First, check you can interrupt the boot sequence at the serial prompt using a MiniUSB cable. For this, I used minicom which I find a fairly easy and straightforward command line serial interface. The only issue I had here was that every time I powered the SheevaPlug up it changed its port, hopping between /dev/ttyUSB0 and /dev/ttyUSB1 – my only advice here is that rather than chasing it around like I did, just reboot it an extra time to get it onto the appropriate port (I used /dev/ttyUSB0 ).

Once you’re connected (you have to be quite quick, to catch it, I found) follow Martin’s advice to check if you need to upgrade the U-Boot. I didn’t, and if you’re in a similar situation to me (ie. bought the Sheevaplug with another OS Install already on SD Card) it’s unlikely you will either, but best to check.

Prepare the Environment – Ensure the plug will boot the Debian Kernel:
#~ setenv mainlineLinux yes
#~ setenv arcNumber 2097
#~ saveenv
#~ reset

This will restart the device. Use dmesg to double-check it’s come back on the same port.

Load the Installer – Replace x.x.x.x with your TFTP Server’s IP Address, and y.y.y.y with whatever IP you’d like your SheevaPlug to take for the sake of the install. Take care to select an IP that is both in the same subnet as your server and doesn’t clash with another device on the network (you knew that anyway, right?):
#~ setenv serverip x.x.x.x
#~ setenv ipaddr y.y.y.y
#~ tftpboot 0x00800000 uImage
#~ tftpboot 0x01100000 uInitrd

The two tftpboot commands can take a little while, and times out if it fails to hit the server more than 20 times (from memory). The fails are shown as T by the TFTP advert. Provided it’s getting some of the data (and isn’t all T’s) then it will, it may just take a couple of goes. It took me two attempts to load the uImage, whereas uInitrd worked first time. Your mileage may vary.

Start the Installer:

#~ setenv bootargs console=ttyS0,115200n8 base-installer/initramfs-tools/driver-policy=most
#~ bootm 0x00800000 0x01100000

At this point, I’m making the assumption that you’ve installed Debian before using the Text Installer. If not, see the Installation Guide.
There’s not much here to trip you up, as the installer has it’s own recommended partition that, reportedly, will work fine. However, for me, I wanted to change my root partition to use BtrFS as its filesystem format, as my messing with the Joggler seemed to suggest it would be a bit more favourable to the Flash Media. If you’re configuring the partition table yourself, please ensure that you create a small (~150Mb) /boot partition with the ext2 filesystem.
I won’t claim to know which is the best filesystem for you. I’m sure the defaults will work just fine. I have also read a few reports that claim ext2 may be better than a journalled filesystem (such as ext3) for the root partition, due to the lack of journalling. I’ll report back if and when my SD Card dies…

Post-Installation – Interrupt the Boot Process one last time…
Interrupt the boot process via serial one more time when the installer finally tells you it’s ready to reboot (on my backup Class 6 card, the installation took nearly 3 hours…) in order to ensure the u-boot is configured correctly to boot Debian automatically. Foolishly, I presumed this would have been the case since it ran Lenny from the SD Card previously, but I was wrong.

#~ setenv bootargs_console console=ttyS0,115200
#~ setenv bootcmd_mmc 'mmcinit; ext2load mmc 0:1 0x00800000 /uImage; ext2load mmc 0:1 0x01100000 /uInitrd'
#~ setenv bootcmd 'setenv bootargs $(bootargs_console); run bootcmd_mmc; bootm 0x00800000 0x01100000'
#~ saveenv

It’s worth reiterating Martin’s notice here. If you mangled the partition table layout yourself and didn’t provide a separate /boot partition, you will need to prefix /uInitrd above with /boot – by default (or if you ensured to create a separate boot partition earlier) this will not be an issue.

And so you should be ready to go. To force the SheevaPlug through its usual boot procedure and boot into your new Debian Squeeze install on SD Card, simply run:

#~ run bootcmd

Early Steps to Reduce the Writes to Flash

There’s a handy page over on the PlugComputer Wiki[2] that details some useful steps and suggestions that may reduce the number of writes to your flash media and, in doing so, extend it’s lifespan. Here’s the pick of those, as applicable to Debian.

Add the following lines to /etc/sysctl.conf:
vm.swappiness=0
vm.laptop_mode=5
vm.dirty_writeback_centisecs=1500
vm.dirty_expire_centisecs=1500

Add the following to /etc/rsyslog.conf:
$ActionFileEnableSync on

It is also reasonable to assume that changing the root filesystem to be mounted with relatime (through /etc/fstab) would be beneficial. I haven’t done this as of this post, as I didn’t want to risk interfering with the work of Flashybrid (below) and haven’t got back to having a look at it. To make this change, edit the appropriate line in /etc/fstab:
rootfs / rootfs relatime,rw 0 0

Installing WView Weather Software

Installing Wview on Debian could not be easier. The Manual has most of the information you need but, in a nutshell, add the following to your /etc/apt/sources.list:

deb http://www.wviewweather.com/apt/lenny lenny main
deb-src http://www.wviewweather.com/apt/lenny lenny main

Then simply #~ apt-get install wview
The one slight annoyance (on my part) here was that the wview package seems to depend on apache2, rather than just recommend it, so it’s going to be installed regardless. Given that I have no intention at the minute of hosting any sites on the Sheevaplug I have no need to even install Apache, but removing it will break the WView dependencies. Rather than mess about with manually editing files and further undo the good work apt does in Debian, I found the easiest solution was just to remove apache from being a startup service, so at least it isn’t using up resources for no good reason.

Easiest way I found to do this was using update-rc.d:

#~ update-rc.d -f apache2 remove

And that’s that. If you decide you want to easily re-enable it run the same command, but drop the -f and swap ‘remove’ with ‘defaults’. Job done.

All that’s left to do now is configure wview to work with your weather station, which can all be achieved by use of a command-line script provided as part of the package. To run, first of all stop wview and then run the script:
#~ /etc/init.d/wview stop
#~ wviewconfig

In my scenario, there was little to change here that I was aware of other than one or two fairly obvious things:

  • Disable the generation of images and HTML templates – until I get around writing my own them for these generated files I have no need for them to be generated, as I won’t be using them.
  • Choose the right code to reflect the Weather Station I’m connecting to it.
  • Set the serial interface location.
  • Set the longitude and latitude (less essential given I’m not doing the autogeneration, but useful to do anyway).
  • Fill out the necessary Wunderground details.

I intentionally left things like the Altitude and Sensor Adjustments at their defaults as they’re set within the weather station hardware and don’t need any extra calculations doing to them. Your own experiences and knowledge (the weather station is my dad’s toy, not mine) may lead you to make different choices.

At this point, you’re ready to start recording your data next time WView starts up, so run:
#~ wviewcleardata
to entirely wipe the existing test data in the WView Database. This process is irreversible so be sure you’re doing it on the right box (if you’re migrating) and that you’re happy to do it.

If you’re migrating your data from a different machine (as I was) or from an earlier version of WView I recommend a glance at the WView Manual for more accurate instructions.

Installing and Configuring Flashybrid

Flashybrid is a neat little utility designed to help reduce the wear-and-tear of frequent write operations to flash media (like the SD card I’ve installed Squeeze to) by mounting the majority of the root filesystem read-only, and being configurable to also mount certain directories of your choosing to RAM to make it writeable for general operation, and sync all your changes back to the media in one go at set times, rather than constantly writing every little change direct to the flash media. Theory is, it’ll lengthen the lifespan of the flash media, which means less messing with reinstalling to new SD cards and the cost associated with that. Time will tell.

Flashybrid is included in the Debian repositories, so installing it is just an apt-get away.
Once installed, you want to make sure it’s set to start in the right order[3]. Following bnms’ instructions in the linked thread, you need to edit the init script that controls Flashybrid (/etc/init.d/flashybrid) to read:

### BEGIN INIT INFO
# Provides: flashybrid
# Required-Start: $local_fs
# Required-Stop: $local_fs
# Should-Start: $local_fs
# Should-Stop: $local_fs
# X-Start-Before: $network
# X-Stop-After: $network
# Default-Start: S
# Default-Stop: 0 6

Once set, you want to remove the existing run level links and get them recreated with the new information:

#~ cd /etc
#~ rm rc{0,1,2,3,4,5,6}.d/*flashybrid
#~ insserv -v flashybrid

Create a /ram directory and now you can worry about the actual program config, which is split into 4 places you need to worry about[4]. Examples are below, as you can see, most of the config files are fairly well documented, and I include them here purely as an example of what I used:

/etc/flashybrid/config
# This is the main configuration file for the flashybrid system.

# The mount point of your flash memory.
FLASHMOUNT=/

# You probably want a ramdisk to be set up with directories that the system
# needs to write data to frequently, so that your flash disk can be mounted
# read-only. Flash memory can only be written to several thousand times over
# its lifetime, so setting up a ramdisk will extend the lifetime of your
# flash a lot. This is the location to mount the ramdisk. Comment out the
# line to not set up a ram disk.
RAMMOUNT=/ram

# Commands to run to enter full mode. The disk will be mounted before
# this.
FULL_CMDS="mount -o remount,rw /"

# Commands to run to enter embedded mode. The disk will be unmounted before
# this. If you wanted to spin a disk down, you could add a hdparm -y
# command.
EMBED_CMDS="mount -o remount,ro /"

# This controls the maximum ammount of memory you want to allocate
# to the tmpfs RAM drive
# This parameter is optional, and if you do not supply if, tmpfs will
# occupy up to 50% of your available memory (ram+swap)
FLASH_MAX=192m

# You may also want to edit the other files in this directory:
# ramtmp, ramstore
VERBOSE=yes

/etc/flashybrid/ramstore
# This is a list of directories that are used to store variable data
# that is written to during normal operation of the system, and that should
# be preserved across reboots.
#
# On boot, the listed directories on your flash disk will be copied to the
# ram disk, and then bind mounts will be used to make the directories on
# the ram disk replace them.
#
# On shutdown, the contents of those directories on the ramdisk will be
# rsynced back to the flash disk.
#
# Directories listed here that are not present will be silently ignored.
# Do not put the trailing slash on directories!

# Should always be in ram disk.
/etc
/var/cache/samba
/var/lib/samba
/var/backups
/var/lib/asterisk
/var/lib/postfix
/var/lib/dbus
/var/lib/logrotate
/var/lib/mysql
/var/webmin
/var/log
/var/run
/root
/home
/var/lib/wview

# If you don't use devfs/udev, you will want /dev in ram, since many things
# need to modify it when the system is running. Not needed on systems
# running udev.
#/dev

# Necessary if the system is to be able to send mail while in embedded

# mode.
/var/spool
/var/mail

/etc/flashybrid/ramtmp
# This is a list of directories that should be set up as temporary
# directories in the ramdisk. On boot, the directories listed will be
# created on the ramdisk, mode 1777. Their contents will not be saved
# across reboots. The equivilent directories on your flash will be
# removed and replaced with symlinks to these directories.

# Do not put the trailing slash on directories!

/tmp
/var/lock
/var/lib/dhcp3
/var/lib/php5
/var/lib/misc
/var/lib/ntp
/var/lib/urandom

# This could go in ramstore if you prefer for its contents to persist

# across reboots.
/var/tmp

/etc/default/flashybrid
ENABLED=yes

And that’s it. As should hopefully be clear, the main point is to ensure that everything you want to be writeable (which is basically the wview directory to enable writing to the SQLite database) to be in the ramstore, which means it gets recorded properly and just requires a fh-sync to write that data back to the SD card. Of course, if the device loses power unexpectedly then any data you’ve not forced a sync of will be lost, which is where the cron jobs set up below come into play.
Of course, that’s a compromise, but it’s one I’ve been aware of since I set out on this route, and to me it’s acceptable. You will have to make up your own minds (but I do recommend getting a UPS if you’re planning to have any device on all the time!).

It’s also worth noting that from this point on (after you next restart) Flashybrid will be active, so any further changes will require a fh-sync in order to make them permanent. Personally, I found it easier just to make all the changes I knew I wanted to make now, and then do a final restart to boot into a fully functional environment.

Cron Jobs and Tidying Up

After a quick restart that really is pretty much it in terms of having a functional system. All that’s left to do now is take the appropriate steps to avoid unnecessary or unacceptable data loss. I opted to do this by virtue of a couple of cron jobs.

First, we need a couple of cron jobs to dump the data from the appropriate SQLite databases to an appropriate location. To that end, I get the following two commands to run late in the night every day:
#~ sqlite3 /var/lib/wview/archive/wview-archive.sdb .dump > /tmp/archivebackup.sql
#~ sqlite3 /var/lib/wview/archive/wview-hilow.sdb .dump > /tmp/hilowbackup.sql

Next, we need to move those away from /tmp to a truly backed up location. As a stopgap, I decided to use a remote little corner of my paid-for web hosting, with FTP to transfer the files to it. About half an hour after the previous cron jobs have run (every night) I use another cronjob to invoke lftp to send them to that webspace and, once done, delete the files.
I used lftp primarily because it supports reading a file with a list of FTP commands for it to run, and then closes, which keeps things simple. This way, every night the previous night’s backups get overwritten by the new ones and provided it works properly, the old backups in /tmp get removed.

It’s obviously not ideal, given that if the Plug dies, I’ll be losing up to a day’s worth of data, but in my scenario that’s acceptable. Ideally, I’d have an iSCSI target on my NAS that could contain the important data to keep the backups at realtime. Alas, I don’t have that luxury just now.

The final cron is one to force a run of fh-sync periodically to write the data back to the SD card. At the minute, I’ve opted for it to be done once a month, but given the UPS, I could probably make that longer.

And the final result? Well, check it out for yourself, over at Wunderground, and at the Capernwray House website.

References

  1. http://www.cyrius.com/debian/kirkwood/sheevaplug
  2. http://www.plugcomputer.org/plugwiki/index.php/Reduce_Flash_Writes
  3. http://plugcomputer.org/plugforum/index.php?topic=1074.msg11987#msg11987
  4. http://forums.plugpbx.org/index.php?PHPSESSID=59e06tsai6m9ttm6v0i2jocij6&topic=32.msg117#msg117

Leave a Reply

Your email address will not be published. Required fields are marked *