Wednesday, July 30, 2014

AFT - an elegant weapon for a more civilized age..

This is a sort of nostalgic post and, in some sense, it is also a "toot your own horn" one as well.  I am writing this mainly for myself.  I am trying to remind myself what I've liked most about programming.

Years ago, actually almost 2 decades ago -- around 1996,  I wrote a text mark up system called AFT.  AFT stood for Almost Free Text. It was inspired by Ward Cunningham's original Wiki mark up but went further.

I had a problem. I didn't like using WYSIWYG word processors and the world was moving towards HTML.  I liked Ward's mark up. He was using it on this new "Wiki Wiki" thing. I answered an invite sent to the Patterns List and became one of the first of the wiki users in 1995.  (But that is a different story or a different time.)

AFT was my attempt at a writing system to produce publishable (web and print) documentation.  Since then, it has developed a (now waning) user base.  You can see how many web pages/documents use it without changing the default "watermark" with this query.

As of Ubuntu 14.04, you can get AFT by issuing an "apt-get install aft" against the standard repository.
I think it is still part of FreeBSD "world".  I believe it still runs under Windows too.

Various "modern" mark up languages (written in "modern" programming languages) have since surpassed AFT in adoption, but for me, it still is a more elegant and pleasurable experience.

Over the years (although not very recently), I've updated, fixed and generally maintained the code.  There are no known crashes (it literally take whatever you throw at it and tries to produce good looking output -- although that may fail) and it doesn't require me to look at the HTML (or PDF) manual  (written in AFT!) unless I want to do something complex.

AFT is implemented in Perl. Originally it was written in awk, but I taught myself Perl so as to re-implement it in the late 1990s.

It is, for me, interesting Perl code.  I have modernized it over the years, but it still doesn't depend on CPAN (a good thing if you just want to have non-programmers "download it" and run without dependencies -- yes I know there are packaging solutions to that problem today...).

AFT has "back end" support for HTML, LaTeX, lout and some rudimentary RTF.  These days I think mostly HTML and LaTeX is used.

You can customize the HTML or LaTeX to support different styles by modifying or creating a configuration file.  This configuration file is "compiled" into a Perl module and becomes part of the run time script.

AFT has been a pleasure to hack on now and then. It still runs flawlessly on new Perl releases and has proven not too fragile to add experimental features to. I've accepted some small code fixes and fragments over the years, but generally it is mostly my code.

As I wrote (and rewrote) AFT, I thought frequently of Don Knuth's coding approach (as excellently documenting in my all time favorite book on programming: Literate Programming).  I certainly can't match the master, but the slow thoughtful development he enthuses was inspiring.

Over the years I've gotten a few "thank you" notes for AFT (but nothing in the past few years) and that makes it my (to date) proudest contribution to Free Software.

Maybe I'll dust off the code and introduce some more experimental features...



Sunday, July 27, 2014

Concurrency and multi-core MCUs (GA144) in my house monitor

My house monitoring system monitors lots of sensors. This suggests a multi-core approach, doesn't it?

The problem with (the current concept of)  multi-cores is that they are typically ruled by a monolithic operating system. Despite what goes on in each core, there is one single point of failure: the operating system. Plus, without core affinity, our code may be moved around.  In a 8 core Intel processor, you are NOT guaranteed to be running a task per core (likely, for execution efficiency, your task is load balanced among the cores).  Each core is beefy too. Dedicating a whole core to a single sensor sounds very wasteful.

This, I believe, is flawed think  in our current concurrency model (at least as far as embedded systems go).

I want multiple "nodes" for computation. I want each node to be  isolated and self reliant.  (I'm talking from an embedded perspective here -- I understand the impracticality of doing this on general purpose computers).

If I have a dozen sensors, I want to connect them directly to a dozen nodes that independently manage them.  This isn't just about data collection. The nodes should be able to perform some high level functions.  I essentially want one monitoring app per node.

For example: I should be able to instruct a PIR motion-sensor node to watch for a particular motion pattern before it notifies another node to disperse an alert. There may be some averaging or more sophisticated logic to detect the interesting pattern.

Normally, you would have a bunch of physically separate sensor nodes (MCU + RF),  but RF is not very reliable. Plus, to change the behavior of the sensor nodes you would have to collect and program each MCU.

So, consider for this "use case" that the sensors are either wired or that the sensors are RF modules with very little intelligence built in (i.e. you never touch the RF sensor's firmware): RF is just a "wire".  Now we can focus on the nodes.

The Green Arrays GA144 and Parallax Propeller are the first widely-available MCUs (I know of) to encourage this "one app per node" approach.  But, the Propeller doesn't have enough cores (8) and the GA144  (with 144 cores) doesn't have enough I/O (for sake of this discussion, since the GA144 has so many cores I am willing to consider a node to be a "group of core").

Now, let's consider a concession...
With the GA144, I could fall back to the RF approach.  I'll can emulate more I/O by feeding the nodes from edge nodes that actually collect the data (via RF).  I can support dozens of sensors that way.

But, what does that buy me over a beefy single core Cortex-M processing dozens of sensors?

With the Cortex-M, I am going to have to deal with interrupts and either state machines or coroutines. (although polling is possible to replace the interrupts, the need for a state machine or coroutines remain the same).  This is essentially "tasking".

This can become heinous. So,  I start to think about using an OS (for task management).  Now I've introduced more software (and more problems).  But can I run dozens of "threads" on the Cortex-M? What's my context switching overhead?  Do I have a programming language that lets me do green threads?  (Do I use an RTOS instead?)

All of this begins to smell of  anti-concurrency (or at least one step back from our march towards seamless concurrency oriented programming).

So, let's say I go back to the GA144. The sensor monitoring tasks are pretty lightweight and independent. When I code them I don't need to think about interrupts or state machines. Each monitor sits in a loop, waiting for sensor input and  a "request status" message from any other node.
In C psuedo-code :

while (1) { 
  switch(wait_for_msg()) {
    case SENSOR: 
       if (compute_status(get_sensor_data()) == ALERT_NOW)
          send_status(alert_monitor);
       break;
    case REQUEST:
       send_status(requester);
       break;
  }
}

This loop is all there is.  The "compute_status" may talk to other sensor nodes or do averaging, etc.
What about timer events? What if the sensor needs a concept of time or time intervals?  That can be done outside of the node by having a periodic REQUEST trigger.

(This, by the way, is very similar to what an Erlang app would strive for (see my previous post GA144 as a low level, low energy Erlang).

Now, the above code would need to be in Forth to work on the GA144 (ideally arrayForth or PolyForth), but you get the idea (hopefully ;-)


Tuesday, July 22, 2014

A Reboot (of sorts): The IoT has got me down. I think we've lost the plot.

The IoT (Internet of Things) has got me down.  I think we've lost the plot.

In most science fiction I've read (and seen), technology is ubiquitous and blends into the background.  The author of a science fiction book may go into excruciating detail explaining the technology, but that is par for the course.

In science fiction films the technology tends to be taken for granted.  Outside of plot devices, all the cool stuff is "just a part of life".

Re-watch Blade Runner, Minority Report, etc. Do the characters obsess (via smartphone or other personal device) over the temperature of their home while they are away?  Do they gleefully purchase Internet connected cameras and watch what their pets are up to?

It is 2014 and we buy IoT gadgets that demand our attention and time.  Nest and Dropcam: I am looking at you.

Beyond "Where is my Jet Pack?", I want "Set and Forget" technology.  The old antiquated "Security Monitoring" services (e.g. ADT) got it partially right. You lived with it. You didn't focus on it and you weren't visiting web pages to obsess over your house's security state.  But that model is dying (or should be). It is expensive, proprietary and requires a human in the loop ($$$).

What do we replace it with?

I think that the "Internet" in the IoT is secondary.  First, I want a NoT (Network of things) that is focused on making my house sensors work together.  Sure, if I have a flood, fire or a break in, I want to be notified wherever I am at (both in the house and out).  When I am away from my home  is where the Internet part of IoT comes into play.

My current Panoptes prototype (based on X10) monitors my house for motion and door events. My wife or I review events (via our smartphone) in the morning when we wake up. It gives me valuable information, such as "when did my teenage son get to bed?" and "was mother-in-law having a sleepless night?" and "is mother-in-law up right now?".  Reviewing this info doesn't require the Internet but does require a local network connection.

I also register for "door events" before I go to bed. This immediately alerts me (via smartphone) if  "mother-in-law is confused and has wandered outside".

When I leave the house, I can monitor (via XMPP using my smartphone) activity in the house. When I know everyone is out, I can register (also via XMPP)  for door/motion events. I can tell if someone is entering my house (our neighborhood has had a recent break in).

This is an important Internet aspect of Panoptes.  I rarely use it though.  My main use of Panoptes turns out to be when I am at home.

So, I want IoT stuff, but I want it to be "Set and Forget".  This is the primary focus in my series of Monitoring projects.