Strategies for Erlang/OTP on a small embedded (turnkey) platform (i.e. Beaglebone)

As I have mentioned before, I am pretty much using Erlang on the Beaglebone as if it was running on a small (very constrained) server. I'm not using it for bit banging or peripheral interfacing.  I've got very low power (and handy) 8-bit micros to deal with that.  Any peripheral I connect to the Beaglebone will be bridged through a UART.

Where Erlang/OTP will work for me on the Beaglebone is to effectively do what it would on a large server:  I want the Beaglebone to be the brains behind my home sensor network.

Why not just use a PC?  Well, this is effectively (once configured) a turnkey system.  Turn it on, stick it onto your network (or if no 24/7 internet connection, connect it to a cellular modem) and it does it's job.  It does it's job with very little power consumption, a small footprint and minimal "boot" time.  It is just another appliance.  (Note: I am doing development and testing on a PC, only every once and a while making sure that the code runs okay on the Beaglebone. But the final target is a Beaglebone.)

However, with this smallness comes some constraints.  The first  is the most glaring one: there is no harddisk. The Beaglebone has a microSD, but that is the boot medium. On a failsafe system we do not boot from the same place as we actively write data.  So, I am going to have to make sure that Erlang logs nothing to the microSD.  (Yes, I know that the microSD is partitioned, but it *is* just one medium. It is one point of failure and I fear the mysterious vendor-dependent machinations of wear leveling).

What? No logs?  Well, think about it.  Who is going to see those logs?  Well, you could upload them to a server on the internet as part of a bug report, right? Really? What use will a bug report be when Grandma's basement is flooding and the monitoring system isn't doing its job?

Okay, this thing is going to have to be reliable. Think "automobile sensor system" reliable.  Think about what happens when your automobile's sensor system fails: This is a very big deal.  Home monitoring doesn't sound as important as your automobile's sensor system, but when it fails the results can lead to similar problems. This is one of the reasons I chose Erlang/OTP.  I want some support (and some nudging) in creating a very reliable home monitoring system.

But, I digress. What are these strategies I must consider when doing Erlang/OTP on such a small platform?

1. No logging to disk. Nope, the microSD is just for booting and maintaining a list of registered sensor nodes with encryption key, signature and assignments.  At some point I am considering adding additional flash storage (NAND serial flash memory?) just for this registration. The microSD needs to be kept clean as possible -- hey I need to check to see what Linux is touching there too (see next item).  In general: Assume you will run 24/7 without total system failures and keep everything in RAM (Erlang ets?). If there is a major glitch requiring a reboot, do it fast and start fresh.
2. Trim Linux.  Honestly, it is pretty cool to be able to ssh into my router or set-top-box.  But, the final version of this home sensor base station should not need a bunch of Linux services running. Trimming services should even improve boot time. I don't want anything running that isn't under *my* control. I'm a bit of a control freak.  Heh, Erlang (beam) could even be process pid 1, as far as I am concerned ;-) Well, I may still want to keep an ssh daemon running to debug it... but it won't be of much use when the box sits at Grandma's house.
3. You don't have a ton of RAM. Run light.  If you are an internet connected device (and my home monitor certainly is), then get the information off of the device (and into the cloud) as soon as possible.
4. One home monitor base station = One single point of failure. Consider a couple of stations (2 Beagle bones?) working together: Redundancy.When both are alive, they agree on which gets to handle "control" (i.e. sending the data to a cloud server, turn on/off lights, etc). They both receive, track and analyze all sensor broadcasts, but only one does anything with that data.  The two servers can be joined by co-heartbeats.  Maybe one has a cellular modem in case the Internet connection goes bye-bye.  This may sound like overkill, but it dramatically increases the reliability of the system.  How to effectively manage this (particularly after a "failover") is tricky. Assuming that the failed server recovers (reboots or is "plugged back into the outlet"), the two need to be brought back into sync (somewhat).

#3 is the most interesting problem right now.  I am hoping that OTP will help me build the solution. I don't need to completely solve it right now, but it should be considered when building my "first, one and only" home monitor base station.  A cluster of Beaglebones anyone?

My Home Monitoring Project: Goals, where/why Forth and Erlang?

A few people have asked, so here is the quick lowdown:

I am building a home sensor network for my house as well as a target group of "independent elderly"  who have grown kids that want to keep track of them (did you leave the stove on after going to bed? etc).  

I would like the system to support lots of distributed sensors, so they need to be inexpensive ($20 target BOM).  Right now, your home sensor choices are X10, Z-Wave, etc.  They are either too power dependent, too limited or too expensive.  You should be able to put a sensor in every room of your house (and maybe some in your garden, garage or yard).

The sensors should be very, very low power and ideally run off of CR2032 batteries or 2 AAAs.  My sensor MCU is a Silabs C8051F912 (8K Flash; 768 bytes of RAM and insanely low power). I am programming it with Charley Shattuck's MyForth (sort of a macro-assembler that feels like Forth). I  am using an RF12B 433Mhz transceiver using my own protocol (soon to be published). The protocol is encrypted using RC4 (with a 3 byte counter -- 16 million unique key sequences before rolling over is secure enough given a 1 sensor message per minute rate; I am mainly using it for authentication to prevent spoofing -- the data itself isn't that "secret").

  I am not that interested in home "control" yet (turning on lights, etc -- anything that involves AC power), but the types of things I want to sense are:

1. Basement flooding (Water activated Switch).
2. Motion (entry/exit of house and rooms).
3. Temperature (each room and outdoors).
4. Stove (Motion + temperature + time-of-day): Is the stove on for for a long time? Is anyone in the kitchen? Is it an odd time for the stove to be on?
5. Soil moisture (garden).
6. Doorbell (button press to ring + log it -- was someone at the door earlier today?)
7. Vibration/Motion (Was someone on the back deck?)
8. Tap detect (did someone knock on the front door)?
9. Open Window/Door

It would be nice to combine as many sensor capability into one device (perhaps vibration, motion, temperature & switch) and then you analyze the data based on what the device is supposed to monitor.

Some of my current sensors elements include:

1. Piezo vibration sensors
2. Passive IR sensors
3. MEMs accelerometers (for knocks, etc)
4. Magenetic reed switches (for detecting open windows and doors)

The base station is currently utilizing a Beaglebone running Erlang (in truth I am doing development on a laptop running Erlang, but my test target is the Beaglebone).  I am also looking at using RabbitMQ for reliable delivery to the "Cloud" (for further processing/notification). A RabbitMQ queue will run on the Beaglebone so if an Internet connection is not available sensor data is locally queued. A local server (also on the Beaglebone) will feed off of the queues to do things like control lights, bells, pumps, etc.  There will be a shovel between the local RabbitMQ and the Cloud based RabbitMQ.

I chose Erlang because it has a nice message protocol parsing capability and OTP is focused on 24/7 availability. I have been familiar with Erlang for over 10 years, but this is the first year I've actually had an opportunity to dive in deeply.  Plus I am doing some Erlang at work so there is some mental synergy.

I chose MyForth because it is nice and forces me to really think about small system development. 

I chose the Silabs 8051 because it is a very, very nice 8-bit family with a ton of peripheral support -- it is also cheap, ubiquitous and low power .  I've used it (and MyForth) on a couple of job related tasks and am very happy with the match.

Where am I right now?

1. One prototype PIR + temperature sensor is complete
2. One base station radio transceiver -> USB/UART prototype completed.
3. Almost done with Erlang base station message processing module (decryption tonight!)

I have a long road ahead, but I am enjoying tackling things at a lower level (I just finished implementing RC4 in MyForth!).

Erlang on Beaglebone: Don't sweat the small stuff

My interests regarding Erlang on the Beaglebone (or any other small low power ARM system) is less about the peripheral device capability and more about running "big system" stuff on small platforms.

I've thought about hooking SPI (or I2C) peripherals to the Beaglebone, but it just seems too complicated to be worth the effort.  Apparently, I will need to patch the Linux kernel and then convince Erlang to play with it by writing a Port driver.   It's really at the Linux level where things start to get complex:

Every peripheral uses SPI in its own manner.  Being synchronous, you just can't ask a SPI device to "get 1024 bytes" and just consume data.  As a master, you need to send a byte for every byte you want to receive (and understand that for 4-wire SPI you will receive a byte simultaneously during the sending of a byte).

You can implement a peripheral's SPI protocol in kernel or user space under Linux.
However, the notion of I/O buffering and time sliced multitasking is somewhat counter to what SPI is about. (Of course you can do SPI in a multitasking environment, but unlike asynchronous UART buffering, if you write a user space SPI driver, you can't expect the kernel to do more than 1 byte worth of  SPI  work for while you wait for your next slice.)

A kernel based SPI protocol is more efficient, but it means that you are mucking about with kernel development for *every* peripheral you want to support.  I don't want to write kernel drivers for every SPI device.

Alas, handling all of this is trivial with a simple microcontroller.

I believe that dealing with low level synchronous protocols is not a good fit for Erlang. I prefer to have Erlang modelling my application domain than worry about bit banging.

My current choice for a home sensor transceiver is a RFM12B. It talks SPI.  It has a 2 byte nternal buffer for bytes it receives over-the-air.  You need a very responsive SPI driver or you will lose data.

I will use a $3 microcontroller as a bridge between the RFM12B and the Beaglebone UART.  The microcontroller will handle the SPI and stream it to a simple asynchronous serial protocol for Linux to receive and buffer.

The Erlang on the Beaglebone will handle the sensor protocol parsing via a standard Linux UART (/dev/XXX).  Because Erlang is adept at parsing binary, I'll keep the UART protocol binary.

I won't use Linux/Erlang to sweat the small stuff -- microcontrollers make excellent bridges!