Create a robust Raspberry PI setup for 24×7 operation

Updated: 9th March 2016 – adapted to latest raspbian 2016-02-26-raspbian-jessie

If you’ve tried to leave a Raspberry PI working 24×7 with the default setup of Raspbian, chances are that at some point (especially after a power outage) it stopped working. Probably the SD card is broken, usually beyond any kind of fix: you have to buy a new one.

However, there’re a few tricks you can use to make a Raspberry PI work 24×7 and recover from a power outage as if nothing had happened.

That’s what I will discuss in this post, hope you find it useful.

Basically, we will make file systems read only and disable swap, so there’s nothing writing to the SD card “without control”. You will be able to still write to the SD, but just when you want to do and it in a controlled way.

But that means we have to fix quite a few things to make things work.

First steps

I used to install with NOOBS, and you can still do it that way, however I now prefer the slightly harder way of using the raspbian image and writing it to the sd with dd (Linux / Mac). You can find detailed instructions in the guides of the Raspberry Pi website.

I prefer this way because there isn’t an option with NOOBS now to reserve part of the SD for a data partition, it will automatically expand the root file system to use all the available space. Using the image it will just create a /boot file system and the root file system using around 4Gb (raspbian) or 1.3Gb (raspbian-lite). The rest of this guide asumes this option, so be careful because the file systems described might not match your system if you use NOOBS (and you should check them in any case).

For this kind of setup I try to get the most RAM available, we won’t use the GUI, so I set VRAM to 16Mb using raspi-config, but let’s get to the real work.

You should make apt-get update & apt-get upgrade.

You might remove now packages that you don’t use (I don’t use wolfram-engine ant it takes a substantial amount of space), or even better: use the lite image and add what you need.

Now check what is in you /etc/fstab file, it should be something like:

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults       0       2
/dev/mmcblk0p2  /               ext4    noatime,defaults  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that

That means that your SD cards is /dev/mmcblk0, you have partition 1 of type vfat mounted as /boot, and partition 2 of type ext4 mounted as root.

Create a data partition

This is an optional step, but I prefer to minimize chances of the root file system being corrupted, so we’ll now create a data partition where we will store our project files.

Well, now write the following command: fdisk /dev/mmcblk0 and then type the p command to show the partition table. You should get something like this (32Gb SD card, raspbian standard):

$ sudo fdisk /dev/mmcblk0
Command (m for help): p
Disk /dev/mmcblk0: 29,8 GiB, 32010928128 bytes, 62521344 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0cf63fa8

Device         Boot   Start     End Sectors  Size Id Type
/dev/mmcblk0p1         8192  131071  122880   60M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       131072 7870463 7739392  3,7G 83 Linux

Type n to create a new partition:

Partition type
   p   primary (2 primary, 0 extended, 2 free)
   e   extended (container for logical partitions)

We can use a primary partition, so press p (you can use an extended one if you wish).

Selected partition 4
First sector (2048-62521343, default 2048): 7870464

Be careful here, it says that the default first sector is 2048 because /boot starts at 8192, but you must use the next free sector after the root partition, in this case 7870464.

Now we choose the size, if we hit enter it will use all available space (recommended option), but we can set a specific size easily (500Mb here):

Last sector, +sectors or +size{K,M,G,T,P} (8894464-62521343, default 62521343): +500M

If you type p to print the partition table it should look like:

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192   131071   122880   60M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       131072  7870463  7739392  3,7G 83 Linux
/dev/mmcblk0p3      7870464  8894463  1024000  500M 83 Linux

Ok, type w to write and exit. You’ll probably need to reboot at this point:

sudo shutdown -r now

After rebooting you need to format the partition, we’ll use ext4:

mkfs.ext4 /dev/mmcblk0p3

Let’s create a mount point for our data partition:

sudo mkdir /data

And create also a folder that will need afterwards:

sudo mkdir /var/spool/mqueue

Now lets change /etc/fstab using our favorite editor, it should end up like:

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    ro       0       0
/dev/mmcblk0p2  /               ext4    noatime,ro  0       0
/dev/mmcblk0p3  /data           ext4    noatime,ro          0       0
tmpfs    /var/log    tmpfs    defaults,noatime,nosuid,mode=0755,size=50m    0 0
tmpfs    /var/lib/sudo    tmpfs    defaults,noatime,nosuid,mode=0755,size=2m    0 0
tmpfs    /var/spool/mqueue    tmpfs    defaults,noatime,nosuid,mode=0700,gid=12,size=30m    0 0
tmpfs   /tmp    tmpfs   defaults,noatime,mode=1777,size=30m

So, we changed boot and root partitions to readonly (ro) and removed the chkdsk option. This last is important because sometimes, as the PI does not have a hardware clock, after reboot it will determine that a file system has changes in the future and wait for a manual fsck to be confirmed. That’s not what we want for our resilient setup.

We added the partition 3 mounted at /data also readonly.

And we created mount points using tmpfs (RAM file system) for several programs and processes that need to write to the filesystem to work properly. You can change the maximum amount of RAM used for that if you want, but remember that the size is the maximum, it will use only the RAM as it needs it, up to that maximum. If a mount point is empty, it will use almost no RAM.

We will now apply a little trick to save RAM, by pointing /var/tmp to /tmp:

$ cd /var
$ sudo rmdir tmp
$ sudo ln -s /tmp .

That way /tmp and /var/tmp go to the same ram filesystem.

Ok, now a few tweaks to make everything work as expected.

Prevent fsck from running at boot:

sudo tune2fs -c -1 -i 0 /dev/mmcblk0p2
sudo tune2fs -c -1 -i 0 /dev/mmcblk0p3

Remove temporary fake-hwclock files:

Edit /etc/cron.hourly/fake-hwclock and put an exit 0 command at the beginning.

Logrotate configuration

Log rotate needs a place to keep track of its state to work, so we need to modify /etc/cron.daily/logrotate, adding –state /var/log/logrotate.state to the command line, so it ends up like:

#!/bin/sh

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate --state /var/log/logrotate.state /etc/logrotate.conf

Now we have to remove the swap:

$ sudo update-rc.d -f dphys-swapfile remove
$ sudo swapoff /var/swap
$ sudo rm /var/swap

Finally, we disable man indexing:

Edit /etc/cron.weekly/man-db and /etc/cron.daily/man-db and put an exit 0 command as the first line.

And that’s it.
You can now reboot your PI:

sudo shutdown -r now

Writing when you need it

Ok, I’ve got my PI up and running, but I need to update it, add new programs, data, How do I do that?
That’s pretty easy!
You need to remount the filesystem you want to modify for writing. For example, to write something on /data:

sudo mount -o remount,rw /data

When you are finished just type:

sudo mount -o remount,ro /data

Sometimes though, there’s some process preventing the filesystem to go back to readonly state. If that’s the case, you need to reboot your PI.

Some considerations

As pointed out in the comments by PrimitiveDesn, there’re other options for swap and logs.

You can use an small thumb drive to store logs and swap.

Also, depending on your setup you might be able to remove local logs and use syslog to store logs on a different machine in the same network.

Thanks to

PrimitiveDesn, gauthma and allen for their comments and fixes.

References

I did search a lot to make this possible, and a lot of credit goes to these pages and their authors:

http://www.a-netz.de/2013/02/read-only-root-filesystem/
http://www.raspberrypi.org/forums/viewtopic.php?f=29&t=22596
http://www.element14.com/community/message/67167/l/re-how-do-i-permanently-disable-the-swap-service#67167
http://www.cyberciti.biz/faq/linux-unix-bypassing-fsck/

Please, leave your thoughts and opinions!

Advertisements

Opening

I will use this first post to establish some disclaimers and state some points that should be taken into account by any reader that gets here.
First of all, despite the fact that I feel really comfortable writing in English, and that most of what I read nowadays is in English, my native language is Spanish, so I’m sure you will be kind and forgive me if I make some mistakes. In fact, you’re welcome if you feel like making a comment on it so I can improve my English 🙂
Last, but not least, I want to thank you for spending your time reading my blog. You’re always welcome to comment, as long as you keep a reasonable language and attitude.