A Song of Sauce and Structure

The Foe

Bluetooth Low Energy firmware applications have their inherent complexities. They’re structured as event-driven systems. That basically mean they react to “events” that can occur at any time.  That’s physical event sources like button presses, battery alerts, and sensors inputs, but also to incoming Bluetooth communications from an App. In our experience, even top-tier chip vendors like Nordic Semiconductors provide subpar guidance on how to best orchestrate this system of events and reactions.
‍‍
An unsuspecting developer may feel they’re using a lean approach, linking event sources in code to their corresponding reactions in a piecemeal fashion. The complexity of this patch network soon grows exponentially and starts feeling like a firmware savant is required just to make sense of it all. As the project grows, the event-reaction mappings grow in quantity and likely require being dynamically re-mapped in different modes of operation. If left untreated, the affliction can get the best of you, spreading faster than you add features. If you’re anything like us, there’s an inevitable face-the-music moment where this event-driven spaghetti code can no longer be tolerated.
‍
This isn’t the glutinous spaghetti that you bite with your teeth. It’s the kind that compiles into bytes and bites to deal with. If you’re not aware of spaghetti code and its nauseating effects, then let’s briefly recap.  “Spaghetti code” is a slang term for code with convoluted structure, so much that it’s difficult to debug and and maintain it.

This isn’t the glutinous spaghetti that you bite with your teeth. It’s the kind that compiles into bytes and bites to deal with. If you’re not aware of spaghetti code and its nauseating effects, then let’s briefly recap.  “Spaghetti code” is a slang term for code with convoluted structure, so much that it’s difficult to debug and and maintain it.

‍
What problems are caused by event-driven spaghetti code?

1. BUGS
   ‍
   The worst kind of bugs. It’s the ugly should-have-been-avoidable bugs that crop up and continue to crop up when you lose a handle on the structure of your application.
   ‍
2. Code duplication
   
   The completely unnecessary there-must-be-a-better-way kind of duplication. When the system for handling events is poorly structured, developers struggle to repurpose code across similar events and states of operation. For example, a “battery charging” and “charge complete” state should be able to elegantly share a lot of charging-related functionality.
   ‍
3. Loneliness
   
   Loneliness, also known as staffing problems due to your unmaintainable codebase.  No one likes to work alone, so restructure your application in a way that your coworkers and future coworkers cherish.
   ‍

Fret Not

I want to recommend a system that’s worked for me — two steps for developers like us to immunize against the spaghetti affliction right from the start.

Overview of the Steps to Avoid Spaghetti Code:

1. Atomic event queue
   ‍
   Rather than reacting to events all over your different interrupt execution contexts, an event queue allows you to consolidate those portions of your code into one place. It’s your conduit for “deferring” the handling of events from many contexts (like interrupts) into one main context. It additionally solves the issue of thread/interrupt safety, by avoiding concurrent access of shared resources across different contexts..  For example, you don’t want a SPI peripheral used in two contexts that can interrupt one another. Instead, both contexts can queue events for the main function to perform the SPI operations sequentially.
   ‍
2. HSM
   
   Use a hierarchical state machine (HSM), and a framework to implement one. An HSM is the  knockout punch for our spaghetti foe. It provides an intuitive and deterministic scheme to define how events should be handled throughout various modes of operation. HSMs are “hierarchical” in the sense that each state can be configured to inherit behaviors from a “superstate”, a more general state that encapsulates it and potentially other states. The hierarchical aspect of HSMs make them far more practical for real-world firmware applications than a textbook flat state machine. A “battery charging” and “charging complete” state could both inherit and share a bulk of their code from an “charger attached” state.
   ‍

Immunization
Step 1/2: Take a queue from your vendor

An event queue is the essence of event-driven architecture. If you’re already using one and understand their merit, you may want to jump ahead to the next part.
‍
Firmware events generally arrive in the context of an interrupt, either hardware or software driven. In the case of some dead-simple events, you might get away handling them right there in the interrupt; But, generally that’s bad practice. There’s a common embedded development commandment:

Thou shalt not dilly dally within an interrupt service routine

And last we forget its companion rule:

Respect thine interrupt/thread safety

Immunization step #1 is using the atomic queue to defer event processing to one location in your main context. That keeps us complaint with the embedded commandments, and sets the groundwork for the step #2.

An event-queue paradigm has been around for a while. Many developers would be hard-pressed to develop complex applications without it. You use the queue to take all your events and  funnel them into a single context where they’re processed one at a time. Right off the bat, that gains you cozy-warm assurances of thread/interrupt safety because event handling isn’t strewn across all sorts of overlapping execution contexts.
Courteous silicon vendors usually provide developers with some form of an atomic queue. Nordic provides nRF52 developers with their “app scheduler”. Texas Instruments provides CC2640 developers with their queue module accompanying their TI-RTOS. For our intents and purposes, both of these modules are just different packagings of a functionally-equivalent queue.
Here’s a snippet of generic example code to drive the concept home:

An event-queue paradigm has been around for a while. Many developers would be hard-pressed to develop complex applications without it. You use the queue to take all your events and  funnel them into a single context where they’re processed one at a time. Right off the bat, that gains you cozy-warm assurances of thread/interrupt safety because event handling isn’t strewn across all sorts of overlapping execution contexts.
Courteous silicon vendors usually provide developers with some form of an atomic queue. Nordic provides nRF52 developers with their “app scheduler”. Texas Instruments provides CC2640 developers with their queue module accompanying their TI-RTOS. For our intents and purposes, both of these modules are just different packagings of a functionally-equivalent queue.
Here’s a snippet of generic example code to drive the concept home:
‍

// eventHandler is always executed in the "main" context
void eventHandler(Event event){
    switch(event){
        case TIMER_FIRED:
            perform_periodic_task();
            start_timer();
            break;
        case BUTTON_PRESS:
            react_to_button_press();
            break;
        case BUTTON_RELEASE:
            react_to_button_release();
            break;
        default:
            break;
    }
}

// Interupt handlers
void timerFired_interrupt(void){
    // Add event to the event queue
    // deferring its handling to the main context
    queue_push(TIMER_FIRED);
}

void buttonPress_interrupt(void){
    queue_push(BUTTON_PRESS);
}

void buttonRelease_interrupt(void){
    queue_push(BUTTON_RELEASE);
}

int main(void){
    // Initialize and configure button interrupt
    initialize_buttons();
    initialize_timer();
    start_timer();

    // Main loop    
    for (;;)
    {
        // idle() stays in a low-power state until there's an event
        idle(); 
        while( !queue_empty() ){
            Event event = queue_dequeue();
            eventHandler(event);
        }
    }
}

Immunization
Step 2/2: Stately Behavior

At this point, all of your event processing has been deferred and consolidated into a main context, like the above example’s “eventHandler()” function.  However, the events still need to be handled in a systematic manner. Unlike the example in Step #1, most firmware projects will include many more events whose handling changes over time. This step recommends using a hierarchical state machine to cleanly implement event handling that adapts throughout different modes/states of operation.
If you’re developing a product, like say a low-power wearable, then your device surely has states. Most states are clear-cut and easy to define. I’m talking On, Off, low-power shipping mode, connected, disconnected, etc. At its basis, a state machine is used to encapsulate  event-handling behavior for each of these different states of operation.
Furthermore, you may find commonality or “hierarchy” among those states — that’s states that include other states.  HSMs are “hierarchical” in the sense that states inherit behavior from the states that contain them (superstates). This hierarchy allows your code to stay clear and concise, sharing event-handling subroutines between similar states.
‍

Example of states & hierarchy for a Bluetooth LE device

Between the state machine’s consolidation/encapsulation of state behaviors and the hierarchical aspect’s management of complex shared state behaviors, we now have our knockout punch against spaghetti code.
If you find this section to be daunting, then skip ahead to the good stuff at the bottom: A link to our open-source HSM implementation called HTHSM. HTHSM’s Github documentation provides a practical example for implementing an HSM.
If you want to learn more, I recommend looking into Miro Samek. He’s a guru of all event-driven firmware things and articulately presents his case for state machines in his “State Machines for Event-Driven Systems”. Like us, he champions the HSM, boasting its practicality in “Introduction to Hierarchical State Machines”.
All implementations of state machines are not at all equal, however. If you’re ready to implement a worthwhile state machine, below are my battle-tested recommendations.

Example of states & hierarchy for a Bluetooth LE device

1. BUGS
   ‍
   For each state, dedicate a function for the handling of its events. Use a function pointer to reference the active state’s event-handler. When an event occurs, this “active state” pointer is used to call the appropriate handler. When transitioning between states, you can simply change the value of that pointer to the desired target state.
   ‍
2. Godsend
   
   They’re are a godsend. When you transition states, the active state will clean up after itself, and the new state (target) has an opportunity to get settled in. This isn’t a big idea, and is in fact just basic courtesy. If your ON state turned the LED on, by gosh, it should have the decency to turn it off when it’s done. Having well defined Exit & Enter make it easy to write firmware that maintains deterministic behavior throughout execution.
   ‍
3. Anarchists be damned
   
   Anarchists be damned, hierarchy is what your state machine needs to be effective. Similar in concept to ”overriding” in OOP, event dispatching starts in the active state and works its way up through its superstates. Substates have an option to suppress event handling from their superstates.  Dispatching of Exit & Enter events should be smart enough to determine where on the superstate chain they need to be executed (beneath the least-common ancestor). I mirror Miro Samek’s sentiment that without hierarchy, state machines lose much of their practicality.

HTHSM:
Your Open-Source HSM Implementation

Wondering if I’m leaving you to implement your own HSM from scratch? Absolutely not. The Humble Transistor made an open-source HSM framework called HTHSM. We use it internally and are excited to share it with you. You can access examples and the source code at the HTHSM repo on Github.

Closing

I hope you found this post informative and maybe even a bit helpful. If you’re using an HSM in your project, have feedback on this post, or questions in general, feel free to drop me a line at ray@thehumbletransistor.com. I’d be great to hear from you.