Strategies for Erlang/OTP on a small embedded (turnkey) platform (i.e. Beaglebone)

As I have mentioned before, I am pretty much using Erlang on the Beaglebone as if it was running on a small (very constrained) server. I'm not using it for bit banging or peripheral interfacing.  I've got very low power (and handy) 8-bit micros to deal with that.  Any peripheral I connect to the Beaglebone will be bridged through a UART.

Where Erlang/OTP will work for me on the Beaglebone is to effectively do what it would on a large server:  I want the Beaglebone to be the brains behind my home sensor network.

Why not just use a PC?  Well, this is effectively (once configured) a turnkey system.  Turn it on, stick it onto your network (or if no 24/7 internet connection, connect it to a cellular modem) and it does it's job.  It does it's job with very little power consumption, a small footprint and minimal "boot" time.  It is just another appliance.  (Note: I am doing development and testing on a PC, only every once and a while making sure that the code runs okay on the Beaglebone. But the final target is a Beaglebone.)

However, with this smallness comes some constraints.  The first  is the most glaring one: there is no harddisk. The Beaglebone has a microSD, but that is the boot medium. On a failsafe system we do not boot from the same place as we actively write data.  So, I am going to have to make sure that Erlang logs nothing to the microSD.  (Yes, I know that the microSD is partitioned, but it *is* just one medium. It is one point of failure and I fear the mysterious vendor-dependent machinations of wear leveling).

What? No logs?  Well, think about it.  Who is going to see those logs?  Well, you could upload them to a server on the internet as part of a bug report, right? Really? What use will a bug report be when Grandma's basement is flooding and the monitoring system isn't doing its job?

Okay, this thing is going to have to be reliable. Think "automobile sensor system" reliable.  Think about what happens when your automobile's sensor system fails: This is a very big deal.  Home monitoring doesn't sound as important as your automobile's sensor system, but when it fails the results can lead to similar problems. This is one of the reasons I chose Erlang/OTP.  I want some support (and some nudging) in creating a very reliable home monitoring system.

But, I digress. What are these strategies I must consider when doing Erlang/OTP on such a small platform?

1. No logging to disk. Nope, the microSD is just for booting and maintaining a list of registered sensor nodes with encryption key, signature and assignments.  At some point I am considering adding additional flash storage (NAND serial flash memory?) just for this registration. The microSD needs to be kept clean as possible -- hey I need to check to see what Linux is touching there too (see next item).  In general: Assume you will run 24/7 without total system failures and keep everything in RAM (Erlang ets?). If there is a major glitch requiring a reboot, do it fast and start fresh.
2. Trim Linux.  Honestly, it is pretty cool to be able to ssh into my router or set-top-box.  But, the final version of this home sensor base station should not need a bunch of Linux services running. Trimming services should even improve boot time. I don't want anything running that isn't under *my* control. I'm a bit of a control freak.  Heh, Erlang (beam) could even be process pid 1, as far as I am concerned ;-) Well, I may still want to keep an ssh daemon running to debug it... but it won't be of much use when the box sits at Grandma's house.
3. You don't have a ton of RAM. Run light.  If you are an internet connected device (and my home monitor certainly is), then get the information off of the device (and into the cloud) as soon as possible.
4. One home monitor base station = One single point of failure. Consider a couple of stations (2 Beagle bones?) working together: Redundancy.When both are alive, they agree on which gets to handle "control" (i.e. sending the data to a cloud server, turn on/off lights, etc). They both receive, track and analyze all sensor broadcasts, but only one does anything with that data.  The two servers can be joined by co-heartbeats.  Maybe one has a cellular modem in case the Internet connection goes bye-bye.  This may sound like overkill, but it dramatically increases the reliability of the system.  How to effectively manage this (particularly after a "failover") is tricky. Assuming that the failed server recovers (reboots or is "plugged back into the outlet"), the two need to be brought back into sync (somewhat).

#3 is the most interesting problem right now.  I am hoping that OTP will help me build the solution. I don't need to completely solve it right now, but it should be considered when building my "first, one and only" home monitor base station.  A cluster of Beaglebones anyone?