Staged firmware delivery with GitHub Actions

Note: You can find the example code for this article in GitHub in the ElectronVector/f303-staged-delivery repository.

In the previous article on GitHub Actions we looked at how to use self hosted runners to deploy firmware to real hardware.

In this article, we're going to extend this concept further -- flashing firmware onto two different STM32 Discovery boards:

• The main board will get deployments whenever we merge to main.
• The release board will get deployments when we create a new release.

This will allow us to continue to use our main branch as our integration branch. Anything pushed to main will get flashed to the main board for testing.

Now though, we're going to use GitHub make releases off of this main branch. When we do, we'll have the firmware flashed to the release board.

This will allow us to have one board always flashed with the latest code for integration testing -- and another board which has the latest released code.

In this configuration, both STM32 Discovery boards are connected to the same Raspberry Pi which will again act as a GitHub self hosted runner for flashing the code.

Configure the Raspberry Pi to handle both STM32 boards

In order to use two different boards with the Raspberry Pi, we're going to need to install two instances of the GitHub Actions self hosted runner application. Each instance will have exclusive access to only one of the boards.

I'm working in a new repository for this article, so we will need to add two new self-hosted runners to GitHub. To start with the first runner go to Settings -> Actions -> Runners and click the New self-hosted runner button.

How to create a new self-hosted runner.

This will give you the instructions for downloading and installing the action runner on the Raspberry Pi. However, we're not going to install the runner in the recommended location. Instead, we're going to install this runner in a folder named actions-runner-main.

Instructions for installing the self-hosted runner on the Raspberry Pi. Don’t use the recommended installation locations, as we need two different instances on the Raspberry Pi.

Instead, we can download the actions runner application to the home folder:

$ curl -o actions-runner-linux-arm64-2.308.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.308.0/actions-runner-linux-arm64-2.308.0.tar.gz

Then create new folders for both runners:

$ mkdir actions-runner-main
$ mkdir actions-runner-release

And extract the runner application to both locations:

$ tar xzf ./actions-runner-linux-arm64-*.tar.gz -C actions-runner-main
$ tar xzf ./actions-runner-linux-arm64-*.tar.gz -C actions-runner-release

Then we can configure the main runner with command (including the token) from the GitHub instructions.

$ cd actions-runner-main
$ ./config.sh --url https://github.com/ElectronVector/f303-staged-delivery --token <insert your token here>

When it comes time to configure your runner, add it to the default run group, name it main or prod, depending on which one you are installing, and label it main. You can get fancier with these names later, but this is all you need for now.

Also use the default work folder.

~/actions-runner-main $ ./config.sh --url https://github.com/ElectronVector/f303-staged-delivery --token <your token here>

# Authentication

√ Connected to GitHub

# Runner Registration

Enter the name of the runner group to add this runner to: [press Enter for Default] 

Enter the name of runner: [press Enter for r2d2] main

This runner will have the following labels: 'self-hosted', 'Linux', 'ARM64' 
Enter any additional labels (ex. label-1,label-2): [press Enter to skip] main

√ Runner successfully added
√ Runner connection is good

# Runner settings

Enter name of work folder: [press Enter for _work] 

√ Settings Saved.

After configuring main you can also configure the release runner from the actions-runner-release folder. You don't need to re-download, but you will need to create a hew runner on GitHub to get a new token.

Once both self-hosted runners are configured, they should look like this in GitHub:

Main and release self-hosted runners in GitHub. These represent our two different STM32 boards.

The last step in configuring our runners on Raspberry Pi is to tell each runner which ST32 board it should use. For this, we'll use the serial number of the ST-Link embedded on each board.

To get the ST-Link serial number, we can run dmesg on the Raspberry Pi and then disconnect/connect each STM32 USB cable. When we connect each STM32 board, there will be messages showing the serial number of each board. In our case, these are the serial numbers:

main    : 066CFF545150898367185232
release : 066EFF545150898367150943

For flashing we're going to use a new Python script with GDB that reads the ST-Link serial number from a file so it will flash the correct board. The GDB script expects the serial number be to in a file called st-link-serial.txt in the root folder of each runner:

~ $ echo 066CFF545150898367185232 > actions-runner-main/st-link-serial.txt
~ $ echo 066EFF545150898367150943 > actions-runner-release/st-link-serial.txt

Here is the gdb-st-link-flash-and-exit.py script that uses this file:

# Load the serial number of the ST-Link to use from a file, if it exists.
serial_command = ''
try:
    with open('../../st-link-serial.txt') as f:
        st_link_serial = f.read().strip()
        serial_command = '-c "adapter serial ' + st_link_serial + '"'
        print('Using ST-Link: ' + st_link_serial)
except:
    serial_command = ''
    print('No ST-Link serial number configured')

# Start openocd and connect to it directly via pipe.
# Use the selected serial number, if one exists.
try:
    gdb.execute('target extended-remote | openocd ' + serial_command + 
    ' -f interface/stlink-dap.cfg -f target/stm32f3x.cfg'
    ' -c "gdb_port pipe; log_output openocd.log"')
    gdb.execute('load')
    gdb.execute('monitor reset')
    gdb.execute('detach')
except Exception as e:
    print('** Error connecting to target **')
    print(e)
finally:
    gdb.execute('quit')

It uses the adapter serial command to OpenOCD to select the correct ST-Link.

Define the main branch workflow

The main branch workflow is going to build and flash the code on the main board whenever new code is pushed. The workflow is defined in the build-and-flash-main.yml workflow:

name: Build and flash target binary
run-name: ${{ github.event.head_commit.message }}
on:
  push:
    branches:
      - main
jobs:
  build:
    uses: ./.github/workflows/build.yml
  flash:
    needs: build
    uses: ./.github/workflows/flash.yml
    with:
      target: main

Here we use the branches filter so that only pushes to the main branch cause this workflow to run.

If you recall the workflow from the previous article, this workflow may look a bit more terse than before. That is because we have refactored the build and flash workflows into their own workflow files. This allows us to reuse them in multiple workflows -- which we will do for the release board.

The new workflows have to go into the same .github/workflows folder. The build workflow (found in build.yml) is similar to the previous one we used to setup the environment and perform the compilation:

name: Build target binary
on: workflow_call
jobs:
  build-target-binary:
    runs-on: ubuntu-latest
    steps:
      # Cache/restore the entire toolchain so we don't need to download it on every run.
      - name: Cache toolchain
        id: cache-gcc
        uses: actions/cache@v3
        env:
          cache-name: cache-toolchain
        with:
          path: /home/runner/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi
          key: build-${{ env.cache-name }}
      # If the toolchain isn't cached, we'll need to install it.
      - if: ${{ steps.cache-gcc.outputs.cache-hit != 'true' }}
        name: Install toolchain
        run: | 
          curl -O https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu/12.2.rel1/binrel/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi.tar.xz \
          && tar -xf arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi.tar.xz -C /home/runner
      - name: Set toolchain path
        run: echo "/home/runner/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/bin" >> $GITHUB_PATH
      - name: Check out code
        uses: actions/checkout@v3
      - name: Build application
        run: |
          cmake . -B build/Release --toolchain cmake/arm-none-eabi.cmake -DCMAKE_BUILD_TYPE=Release \
          && cmake --build build/Release
      - name: Save target binary
        uses: actions/upload-artifact@v3
        with:
          name: f303
          path: build/Release/f303

There are two notable differences here from our previous version. First, we use the on: workflow_call trigger which allows us to call this from another workflow file.

The other difference is that we use the actions/cache@v3 action to cache the build toolchain between workflow runs. This means we don't have to download and extract the toolchain for every workflow run. Instead whatever we tell GitHub to cache (the entire extracted toolchain folder in this case) will be saved and restored in any subsequent workflow run. This ends up reducing the build time workflow from 30s to around 10s. For more information refer to the GitHub documentation on caching dependencies to speed up workflows.

The flash workflow takes an input argument which we use to tell the workflow which device to flash (as identified by the label we gave the self-hosted runner).

To tell it to flash the main board, set the target input to main with this configuration:

with:
    target: main

In the flash.yml workflow, this is accessed with ${{ inputs.target }}:

name: Flash target
on:
  workflow_call:
    inputs:
      target:
        required: true
        type: string
jobs:
  flash-target:
    runs-on: [ self-hosted, "${{ inputs.target }}" ]
    steps:
      - name: Check out code
        uses: actions/checkout@v3
      - name: Download target binary
        uses: actions/download-artifact@v3
        with:
          name: f303
      - name: Flash target
        run: gdb-multiarch -x gdb-st-link-flash-and-exit.py f303

The runs-on configuration is how you tell GitHub which self-hosted runner to use. The following configuration:

runs-on: [self-hosted, main]

will instruct GitHub to use the self-hosted runner (the ones on our Raspberry Pi) that has been labeled main.

Drafting releases -- an intermediate workflow

Our ultimate goal with the release board is that we want it flashed whenever we do a release from GitHub. Creating a release can be automated with GitHub actions, but I like the release process to have a manual trigger so that a human can:

1. Decide when to make a release.
2. Update the release notes.

There are steps that we can automate however, to reduce the number of manual operations and to make the release process less error prone. One way to do this is to automate the creation of a "draft" release.

A draft release is just a release that hasn't yet been published. By starting with a draft release, we'll be able to automate a few things and still review and update the release notes before publishing.

The draft release workflow is manually triggered and is responsible for:

• Creating a tag for the release.
• Creating the draft release.
• Setting a release version number.
• Adding the target binary.

To create a manually triggered workflow, we use the on: workflow_dispatch trigger. When this trigger is configured for a workflow it gets a Run workflow button used to start it. Note that the workflow_dispatch action must be on the default branch (main in our case).

The Run workflow button is used to manually trigger an action.

This workflow is defined in draft-release.yaml:

name: Draft release from main
run-name: Draft release ${{ inputs.version }}
permissions:
  contents: write
on:
  workflow_dispatch:
    inputs:
      version:
        description: 'New release version number'
        required: true
        type: string
jobs:
  build:
    uses: ./.github/workflows/build.yml
  create-release:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Create tag
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.git.createRef({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: 'refs/tags/${{ inputs.version }}',
              sha: context.sha
            })
      - name: Download target binary
        uses: actions/download-artifact@v3
        with:
          name: f303
      - name: Create release
        uses: softprops/action-gh-release@v1
        with:
          name: ${{ inputs.version }}
          tag_name: ${{ inputs.version }}
          draft: true
          files: f303

This workflow creates a version name as input which is used to title the release and name the tag that is created. When clicking the Run workflow button there is a field to enter the version name:

When an input is required for a manually triggered workflow, there is field provided where it can be entered.

Creating the release workflow

The last workflow that we need is the one that will flash the release board when a draft release is published. This is defined in flash-release.yml:

name: Flash published release
run-name: Flash release ${{ github.ref_name }}
on:
  release:
    types: [published]
jobs:
  flash-target:
    runs-on: [ self-hosted, release ]
    steps:
      - name: Check out code
        uses: actions/checkout@v3
      - name: Download target binary
        uses: dsaltares/fetch-gh-release-asset@master
        with:
          repo: 'ElectronVector/f303-staged-delivery'
          version: 'tags/${{ github.ref_name }}'
          file: 'f303'
          token: ${{ secrets.GITHUB_TOKEN }}
      - name: Flash target
        run: gdb-multiarch -x gdb-st-link-flash-and-exit.py f303

This action runs on only published releases, and we set it to run on the [ self-hosted, release ] runner which will target our release board for flashing.

Putting it into action

Now that we have all three actions in place, we can walk through how to use them to flash our code to our boards in a staged way.

The first step is to push a change to the main branch which will build and flash the code onto the main board.

The main workflow runs on any push to the main branch — and builds the code and flashes the main board.

Now that there is some code on main, we can start the release process by "drafting a release" with the draft release workflow. On the Actions page, select the Draft release from main workflow and click the Run workflow button. Enter a name for the release and click Run workflow.

Drafting a release.

This will run the draft release workflow:

The draft release workflow has run.

After the workflow has completed, there will be a new draft release in the repository. Navigate to the releases by clicking Releases on the Code tab:

Finding the releases in a GitHub repo.

The new release is listed on the releases page. To publish the release, we can click on the edit icon:

The releases page.

This brings us to the page where we can edit the release. We could add whatever release notes we want in here and when we're ready we click Publish release:

Publishing a release.

When the release is published, it triggers the workflow to flash our release board:

Flash the release image onto the board.

And there we have it! We have our workflows set up to automatically flash our main board whenever we integrate new code on the main branch.

When we have accumulated enough work on main and we've tested it for confidence, then we can decide to do a release. Only after completing the release of the code is the firmware on the release board updated.

Now we can continue to integrate new features on the main branch (and testing them on the main board) while the release board still has our most recent released code. This is useful for testing the and working with the two different builds at the same time.

Continuous firmware delivery with Github Actions

How do you get your firmware builds to your test team?

(Hopefully you have a test team… or at least someone independent who can test your code.)

Do you build locally on your laptop and then email a file? Do they know how to flash it on the device… and have the hardware they need? Do they know how to connect it correctly?

Recently one of my firmware teammates built a (super neat!) browser-based tool. At one point it wasn’t working for me and it needed a fix. So he made the change and I refreshed the browser… and in a matter of seconds the tool was working for me.

I said to him: Wow, that’s a lot easier than deploying firmware, huh?

I didn’t have to download, install, or flash anything. I just refreshed the browser and it was there.

This sort of thing is generally pretty easy (and common!) for web applications, because they’re centrally hosted and can be updated for everyone just by changing the files on the server.

Getting new firmware onto your embedded system is usually more complicated and requires specialized knowledge and hardware.

IDEs often provide this functionality (press F5 to build and load!) but that only works for your local copy of the code with your device connected to your machine.

How do you get this code into devices where other people can use and test it?

The answer is continuous delivery (CD).

CD is the process that automatically delivers our firmware to test systems when we push our code to source control. No manual steps are needed – the code is just there.

Note here that I’m talking about continuous delivery and not deployment. To me, deployment means making the code available to users (like on a website, or to an in-field product). That’s the sort of thing that could be handled by an over-the-air update (if we have one) – but could also be integrated with the CD system!

The plan

If you are using Github for source control, it’s fairly straightforward to add a workflow to deploy your code to real hardware. For the rest of this article, we’ll look at how to set up a very basic example of such a system. It uses just a few components, shown here:

In this example we’ll configure Github actions (by defining “workflows”) to run when we push code to our repository.

Workflows are defined by adding special files to our repository which include 1) commands to execute and 2) which “runner” they should be executed on. Typically, runners are short-lived (cloud hosted) containers  – e.g. an Ubuntu instance – created by Github as needed for your workflows.

In this case, first the application will be built in a cloud hosted runner. Then we’ll use a “self-hosted runner” on a Raspberry Pi to do the flashing on the target board (an STM32F303 discovery board here). The self-hosted runner allows us to use Github Actions to define workflows that run on our own hardware.

We’re using a Raspberry Pi in this example only because it’s small, inexpensive, and convenient. We could use any of the platforms supported by the Github runner application (Linux, Windows, or macOS). This application doesn't require any specialized hardware – just a USB port that we’ll use to connect to our programmer/debugger.

In this example, we’ll use an STM32F303 Discovery board as our target. Our actual target doesn’t matter, but this board is fairly common (and I happened to have one here) and it has an integrated ST-Link that we can use with OpenOCD on the Raspberry Pi for flashing.

The application that we’ll use for testing here is a simple LED blinky that has been adapted to run on the STM32F303 Discovery board.

Configure the Raspberry Pi to flash the target

The STM32 Discovery board has a built-in ST-Link programmer that we can use for flashing via USB from the Raspberry Pi. In this case, we’ll use both OpenOCD and GDB – so we’ll need to install them on the Raspberry Pi.

Install GDB with:

sudo apt install gdb-multiarch

Install OpenOCD with:

sudo apt install openocd

Note: By default, the ST-Link is assigned to the plugdev group when connected. You need to add your Raspberry Pi username to plugdev to have permission to access the ST-Link:

sudo usermod -a -G plugdev <your-rpi-username>

There is a GDB script file in the repo (gdb-st-link-flash-and-exit) that can be used to flash the board with an elf file produced by our build process. The script starts OpenOCD (which starts a GDB server) and then connects to it, loads the file, resets the board, and exits. The script looks like this:

# Start openocd and connect to it directly via pipe

target extended-remote | openocd -f interface/stlink-dap.cfg \
    -f target/stm32f3x.cfg \
    -c "gdb_port pipe; log_output openocd.log"

load
monitor reset
quit

You use this to flash the target board from the Raspberry Pi with:

gdb-multiarch -x gdb-st-link-flash-and-exit <elf-file-name>

Create a workflow to build the firmware

Before we can flash the board, first we need to build the application. We want this build to happen automatically every time we push code to the repo. To do this, we’re going to define a workflow in Github Actions.

The workflow will go in the repository in the .github/workflows folder, in a file named build.yml. The .github/workflows folder is a special location where Github looks for workflows to execute. The YAML workflow file can be named whatever we want – and we can have many workflows defined in many different files.

The workflow is a YAML file that must define a few different items for the workflow to run correctly:

• The name of the workflow.

• When to run this workflow – here it’s configured to run on every push to the repo.

• A job that will do the building.

• The type of environment in which to run the workflow – in this case, we’re running it on the latest version of Ubuntu.

• Install the toolchain (cross compiler).

• Check out the code.

• Build the application.

• Upload the target file to Github.

name: Build target binary
on: [push]
jobs:
 build:
   runs-on: ubuntu-latest
   steps:
     - name: Install toolchain
       run: |
         curl -O https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu/12.2.rel1/binrel/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi.tar.xz \
         && tar -xf arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi.tar.xz -C /opt \
         && echo "/opt/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/bin" >> $GITHUB_PATH
     - name: Check out code
       uses: actions/checkout@v3
     - name: Build application
       run: |
         cmake . -B build/Release --toolchain cmake/arm-none-eabi.cmake -DCMAKE_BUILD_TYPE=Release \
         && cmake --build build/Release
     - name: Save target binary
       uses: actions/upload-artifact@v3
       with:
         name: f303
         path: build/Release/f303

Now, if you push this file to the repository you will see this action start running. When it’s done the f303 build artifact (elf file) will be available to download from Github.

To see your workflows, click the Actions tab at the top of the repo page.

Clicking on one of the workflow runs will take you to the details page, where your can download the build artifact:

You can also click on the the build job allows you to inspect all of the output from every step of the workflow run:

Add a new self-hosted runner

Warning: Github highly recommends that you do not use self-hosted runners on public repositories as this can be a security risk – allowing forks of your repository to use your self-hosted runner to execute potentially dangerous code on your runner machine.

Now that the firmware is building in the Github Actions workflow, we just need to setup a workflow to run on the Raspberry Pi and flash the target. The first step is to add a new self-hosted runner to the repository – this is the one that we are going to host ourselves (on the Raspberry Pi).

To add the self-hosted runner, click on the Settings tab at the top of the repo page, then select Actions and then Runners. Finally, click the New self-hosted runner button in the upper right:

From here, you need to select the OS and architecture of the target. I’m actually using a 64-bit installation on my Raspberry Pi 4, so I’ve selected the ARM64 architecture. This page also conveniently gives us all the instructions we need to install the runner on our Raspberry Pi.

Note: I’ve redacted the token that Github provided for me here. When you set this up you’ll get a token which you’ll use to connect your self-hosted runner to the repository.

Install the runner on the Raspberry Pi

Now we just need to follow the steps Github just provided above to install the runner application on the Raspberry Pi. When you start it, it should look like this:

Modify our workflow to flash the target from the Raspberry Pi

Now that we have the Github runner application installed and running on the Raspberry Pi, we can update our existing build.yml workflow to include a new job to do the flashing of the target.

First we’ll want to rename the workflow file from build.yml to something like build-and-flash.yml (naming is important!) to indicate its expanded functionality.

Then we’ll want to add a new job to the workflow file that will execute the flashing commands on the Raspberry Pi. This job requires just a few configuration items:

• Configure the job to run on the self-hosted runner.

• Configure the job to run only after the build is complete (be default jobs would be run in parallel).

• Check out the code. We actually only need the code in this case to get the gdb-st-link-flash-and-exit command file that we’ll use with GDB to do the flashing.

• Download the binary file saved during the build job.

• Flash the target with GDB.

The entire build-and-flash-workflow.yml will look like this:

name: Build and flash target binary
on: [push]
jobs:
 build:
   runs-on: ubuntu-latest
   steps:
     - name: Install toolchain
       run: |
         curl -O https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu/12.2.rel1/binrel/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi.tar.xz \
         && tar -xf arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi.tar.xz -C /opt \
         && echo "/opt/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/bin" >> $GITHUB_PATH
     - name: Check out code
       uses: actions/checkout@v3
     - name: Build application
       run: |
         cmake . -B build/Release --toolchain cmake/arm-none-eabi.cmake -DCMAKE_BUILD_TYPE=Release \
         && cmake --build build/Release
     - name: Save target binary
       uses: actions/upload-artifact@v3
       with:
         name: f303
         path: build/Release/f303
 flash:
   runs-on: self-hosted
   needs: build
   steps:
     - name: Check out code
       uses: actions/checkout@v3
     - name: Download target binary
       uses: actions/download-artifact@v3
       with:
         name: f303
     - name: Flash target
       run: gdb-multiarch -x gdb-st-link-flash-and-exit f303

After pushing this code, the build-and-flash workflow will show two different jobs (both of which can be inspected independently).

When this workflow run is complete, the code has been built and flashed onto the target successfully.

In summary

This is a very basic example of how to use cloud-based Github Actions to flash real hardware. This makes it possible to deliver firmware builds – as soon as new firmware is pushed – to whoever might need it. This could be useful for internal testers that want to start as new firmware is ready.

Next steps

Note however that there are a lot of things that we didn’t do in this example. Depending on the situation, to make this more useful we might do things like:

• Configure the build and flash workflow to run only on pushes to a certain branch, e.g. some sort of integration branch. This would make it easier for multiple developers to work together.

• Configure different types of target boards, with different hardware. This could be useful if there have been a few different revisions of the target hardware. Loading automatically on all the different revisions would make backwards compatibility testing much easier.

• Configure multiple target boards – each to be flashed from pushes to different branches. This could allow us to have a development as well as a release branch. This is more aligned with common web development practices.

Additionally, it makes sense to do at least a couple of things to the Raspberry Pi:

• Configure it for additional security. We didn’t cover the Raspberry Pi setup, but it makes sense to be a bit more cautious with security.

• Set up the runner application to run as a service. This will start the service whenever the Raspberry Pi is powered on so it’s always available.

There is also a change we could make to make shorten the workflow execution time. Currently, the Github-hosted runner that does the build has to be reconstructed each time it is run. This includes downloading and installing the toolchain which can take up to 30 seconds. Instead of rebuilding the runner environment for each run, using a pre-built Docker image would require less time.

Sample source code for this article can be found here: https://github.com/ElectronVector/f303-continuous-delivery 

Getting Started with Ceedling [3/3] - Building a release binary

This is the third of a three video series to show how to quickly get started with Ceedling for test driven development in C -- by test driving a simple "FizzBuzz" example.

In the previous video we test drove a simple example -- writing the tests along with the code.

In this video we look at how to use Ceedling to build a release binary (it's pretty easy!).

In this example we used native GCC to compile for our host platform, but you can could the release_compiler setting in the project.yml file to compile for whatever target you want.

I hope you enjoyed this quick intoduction to test-driving embedded software with Ceedling. If you're new to unit testing or test driven developement, Ceedling is a really great way to get started. In just a few minutes we were able to create a new project and develop a high quality, well-tested FizzBuzz implementation.