Updated: 9th March 2016 – adapted to latest raspbian 2016-02-26-raspbian-jessie
If you’ve tried to leave a Raspberry PI working 24×7 with the default setup of Raspbian, chances are that at some point (especially after a power outage) it stopped working. Probably the SD card is broken, usually beyond any kind of fix: you have to buy a new one.
However, there’re a few tricks you can use to make a Raspberry PI work 24×7 and recover from a power outage as if nothing had happened.
That’s what I will discuss in this post, hope you find it useful.
Basically, we will make file systems read only and disable swap, so there’s nothing writing to the SD card “without control”. You will be able to still write to the SD, but just when you want to do and it in a controlled way.
But that means we have to fix quite a few things to make things work.
First steps
I used to install with NOOBS, and you can still do it that way, however I now prefer the slightly harder way of using the raspbian image and writing it to the sd with dd (Linux / Mac). You can find detailed instructions in the guides of the Raspberry Pi website.
I prefer this way because there isn’t an option with NOOBS now to reserve part of the SD for a data partition, it will automatically expand the root file system to use all the available space. Using the image it will just create a /boot file system and the root file system using around 4Gb (raspbian) or 1.3Gb (raspbian-lite). The rest of this guide asumes this option, so be careful because the file systems described might not match your system if you use NOOBS (and you should check them in any case).
For this kind of setup I try to get the most RAM available, we won’t use the GUI, so I set VRAM to 16Mb using raspi-config, but let’s get to the real work.
You should make apt-get update & apt-get upgrade.
You might remove now packages that you don’t use (I don’t use wolfram-engine ant it takes a substantial amount of space), or even better: use the lite image and add what you need.
Now check what is in you /etc/fstab file, it should be something like:
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults 0 2 /dev/mmcblk0p2 / ext4 noatime,defaults 0 1 # a swapfile is not a swap partition, so no using swapon|off from here on, use dphys-swapfile swap[on|off] for that
That means that your SD cards is /dev/mmcblk0, you have partition 1 of type vfat mounted as /boot, and partition 2 of type ext4 mounted as root.
Create a data partition
This is an optional step, but I prefer to minimize chances of the root file system being corrupted, so we’ll now create a data partition where we will store our project files.
Well, now write the following command: fdisk /dev/mmcblk0 and then type the p command to show the partition table. You should get something like this (32Gb SD card, raspbian standard):
$ sudo fdisk /dev/mmcblk0 Command (m for help): p Disk /dev/mmcblk0: 29,8 GiB, 32010928128 bytes, 62521344 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0cf63fa8 Device Boot Start End Sectors Size Id Type /dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA) /dev/mmcblk0p2 131072 7870463 7739392 3,7G 83 Linux
Type n to create a new partition:
Partition type p primary (2 primary, 0 extended, 2 free) e extended (container for logical partitions)
We can use a primary partition, so press p (you can use an extended one if you wish).
Selected partition 4 First sector (2048-62521343, default 2048): 7870464
Be careful here, it says that the default first sector is 2048 because /boot starts at 8192, but you must use the next free sector after the root partition, in this case 7870464.
Now we choose the size, if we hit enter it will use all available space (recommended option), but we can set a specific size easily (500Mb here):
Last sector, +sectors or +size{K,M,G,T,P} (8894464-62521343, default 62521343): +500M
If you type p to print the partition table it should look like:
Device Boot Start End Sectors Size Id Type /dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA) /dev/mmcblk0p2 131072 7870463 7739392 3,7G 83 Linux /dev/mmcblk0p3 7870464 8894463 1024000 500M 83 Linux
Ok, type w to write and exit. You’ll probably need to reboot at this point:
sudo shutdown -r now
After rebooting you need to format the partition, we’ll use ext4:
mkfs.ext4 /dev/mmcblk0p3
Let’s create a mount point for our data partition:
sudo mkdir /data
And create also a folder that will need afterwards:
sudo mkdir /var/spool/mqueue
Now lets change /etc/fstab using our favorite editor, it should end up like:
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat ro 0 0 /dev/mmcblk0p2 / ext4 noatime,ro 0 0 /dev/mmcblk0p3 /data ext4 noatime,ro 0 0 tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0755,size=50m 0 0 tmpfs /var/lib/sudo tmpfs defaults,noatime,nosuid,mode=0755,size=2m 0 0 tmpfs /var/spool/mqueue tmpfs defaults,noatime,nosuid,mode=0700,gid=12,size=30m 0 0 tmpfs /tmp tmpfs defaults,noatime,mode=1777,size=30m
So, we changed boot and root partitions to readonly (ro) and removed the chkdsk option. This last is important because sometimes, as the PI does not have a hardware clock, after reboot it will determine that a file system has changes in the future and wait for a manual fsck to be confirmed. That’s not what we want for our resilient setup.
We added the partition 3 mounted at /data also readonly.
And we created mount points using tmpfs (RAM file system) for several programs and processes that need to write to the filesystem to work properly. You can change the maximum amount of RAM used for that if you want, but remember that the size is the maximum, it will use only the RAM as it needs it, up to that maximum. If a mount point is empty, it will use almost no RAM.
We will now apply a little trick to save RAM, by pointing /var/tmp to /tmp:
$ cd /var $ sudo rmdir tmp $ sudo ln -s /tmp .
That way /tmp and /var/tmp go to the same ram filesystem.
Ok, now a few tweaks to make everything work as expected.
Prevent fsck from running at boot:
sudo tune2fs -c -1 -i 0 /dev/mmcblk0p2 sudo tune2fs -c -1 -i 0 /dev/mmcblk0p3
Remove temporary fake-hwclock files:
Edit /etc/cron.hourly/fake-hwclock and put an exit 0 command at the beginning.
Logrotate configuration
Log rotate needs a place to keep track of its state to work, so we need to modify /etc/cron.daily/logrotate, adding –state /var/log/logrotate.state to the command line, so it ends up like:
#!/bin/sh test -x /usr/sbin/logrotate || exit 0 /usr/sbin/logrotate --state /var/log/logrotate.state /etc/logrotate.conf
Now we have to remove the swap:
$ sudo update-rc.d -f dphys-swapfile remove $ sudo swapoff /var/swap $ sudo rm /var/swap
Finally, we disable man indexing:
Edit /etc/cron.weekly/man-db and /etc/cron.daily/man-db and put an exit 0 command as the first line.
And that’s it.
You can now reboot your PI:
sudo shutdown -r now
Writing when you need it
Ok, I’ve got my PI up and running, but I need to update it, add new programs, data, How do I do that?
That’s pretty easy!
You need to remount the filesystem you want to modify for writing. For example, to write something on /data:
sudo mount -o remount,rw /data
When you are finished just type:
sudo mount -o remount,ro /data
Sometimes though, there’s some process preventing the filesystem to go back to readonly state. If that’s the case, you need to reboot your PI.
Some considerations
As pointed out in the comments by PrimitiveDesn, there’re other options for swap and logs.
You can use an small thumb drive to store logs and swap.
Also, depending on your setup you might be able to remove local logs and use syslog to store logs on a different machine in the same network.
Thanks to
PrimitiveDesn, gauthma and allen for their comments and fixes.
References
I did search a lot to make this possible, and a lot of credit goes to these pages and their authors:
http://www.a-netz.de/2013/02/read-only-root-filesystem/
http://www.raspberrypi.org/forums/viewtopic.php?f=29&t=22596
http://www.element14.com/community/message/67167/l/re-how-do-i-permanently-disable-the-swap-service#67167
http://www.cyberciti.biz/faq/linux-unix-bypassing-fsck/
Please, leave your thoughts and opinions!