Load board configurations dynamically at runtime

[neilenns]

I'd like to change how board configurations are managed in MobiFlight-Connector so they load dynamically from XML files at runtime. Currently they are hardcoded in MobiFlightModuleInfo.cs which makes several scenarios challenging:

1. Adding a new vid/pid combination for existing supported boards requires rebuilding and publishing a new desktop app
2. Adding a wholly new board that knows how to talk MobiFlight's protocol requires spoofing an existing board type
3. Adding boards that have different pin configurations requires a ton of invasive code that may not be appropriate to ship to everyone, thus requiring manual merging of every MobiFlight Connector change.

We've briefly touched on two approaches to doing this in the Discord chat: dynamically load on the desktop or query the device for its capabilities. I believe the simplest approach is to load on the desktop. It scopes the changes to one side of the MobiFlight system, and doesn't require adding large amounts of static data (pin assignments) to an already size-constrained firmware. It also means all existing firmware out there (including custom versions) can take advantage of this without having to make any changes.

To implement this I will:

1. Create a Board class that encapsulates the various configuration options currently hard coded in MobiFlightModuleInfo.cs
2. Create a BoardDefinition class that loads XML files from disk when MobiFlight launches and provides GetBoardByHardwareId() and GetBoardByName() methods that return the appropriate board definition
3. Update places in code that refer to the existing hardcoded classes to use the loaded board definition instead
4. Remove places where tests are made against specific board types (e.g. to detect which firmware to load) and replace it with properties that are stored on the Board class (e.g. the firmware filename, the most recent firmware version, etc.)

[neilenns]

I've started to play around with this and it's sooooooooooooo cool and has so much promise!

[neilenns]

Having said that it's gonna be quite a bit of work. There are tons of places with hardcoded checks for the board type, and all of the board detection logic will need to get re-done to use the dynamic configs and then associate the board config with the detected modules.

[neilenns]

I pretty much have this working at this point. Everything flows through all the way to the module config dialog. Just need to do the grunt work of removing dependencies on the Pin arrays and whatnot.

[MobiFlight-Admin]

hi @neilenns! sounds really cool, but i would really appreciate to discuss things a bit more before going all into it.

Maybe there is a clever way to keep the changes to the info class and not having to change so many places in the code.

[neilenns]

Yeah that basically is the approach I've been playing with, and then it is just cleaning up places that were explicitly checking by device type.

We should chat about input shifters too, are you free tomorrow?

[neilenns]

There was a bunch of discussion in Discord today about the Nano and ProMini. Building the firmware for these is trivially possible now thanks to the PlatformIO migration.

If MobiFlight supported dynamically loading board configurations from an XML file it would only require dropping new config files for the Nano and ProMini into the correct folder and then they would Just Work with MF.

[neilenns]

I've spent a bunch of time digging through the code, understanding how things work today. Here's what I discovered.

How things work today

When MobiFlight runs it needs to know two things:

1. What USB devices are connected that I know how to talk to?
2. What type is each of those devices?

For the first question MobiFlight uses the FriendlyName reported by the USB device or the VID/PID of the device if no friendly name match is found. This check is done in MobiFlightCache.cs:getArduinoPorts().

The second test is done in two different ways. If the MobiFlight firmware is installed then the type is determined based on the value reported from the firmware for MOBIFLIGHT_TYPE. If the firmware isn't installed it falls back to trying FriendlyName and VID/PID.
This test is done (somewhat indirectly) in MobiFlightCache.cs:LookupAllConnectedArduinoModulesAsync().

The key thing to note with this approach is the type mapping based on the firmware reported value is hardcoded and only supports Uno, Pro Micro, and Mega boards. This hardcoded mapping is used throughout the code to make decisions on what is supported by the boards, how to flash devices, etc.

Proposed changes

MobiFlight is fundamentally doing the right checks today: first determine if the connected USB device is supported, then
figure out if the MobiFlight firmware is installed and use the reported type from the firmware to decide what kind of board it is.

My proposal has two parts:

1. Change where the source of valid FriendlyName, VID/PID, and MOBIFLIGHT_TYPE values comes from
2. Create a Board object and associate it with MobiFlightModule to provide other board-specific options

The first part relies on dynamically loading a list of board definitions from XML at startup instead of relying on a hardcoded list. The files contain the expected FriendlyName, valid VID/PID regexes, and MOBIFLIGHT_TYPE that identify the board, as well as all other like the valid list of pins, avrdude parameters, EEPROM size, etc.

The second change is to associate the board definition with the MobiFlightModule created in LookupAllConnectedArduinoModules(). This association makes the entire board definition availble to the rest of the MobiFlight codebase any time the MobiFlightModule is accessed. It makes it trivial to replace all the hardcoded places that used to check module.Type with tests against module.Board.something.

Here's an example of how the code changes once the board definition is available in the module. Today MainForm.cs:CheckForFirmwareUpdates() does the following test to see if the firmware is out of date:

Version latestVersion = new Version(MobiFlightModuleInfo.LatestFirmwareMega);
if (module.Type == MobiFlightModuleInfo.TYPE_MICRO)
    latestVersion = new Version(MobiFlightModuleInfo.LatestFirmwareMicro);
if (module.Type == MobiFlightModuleInfo.TYPE_UNO)
    latestVersion = new Version(MobiFlightModuleInfo.LatestFirmwareUno);

with a board definition available on the module object this simply becomes:

Version latestVersion = new Version(module.Board.LatestFirmwareVersion);

Advantages to this approach

1. There's no change to the firmware code at all. This is entirely desktop side and can be rolled out transparently to all current MobiFlight users.
2. It makes it trivial to add additional Arduino boards. For example adding support for the Nano only requires copying the board definition for the Uno and changing the VID/PID list and MobiflightType.
3. Existing boards that show up with new VID/PIDs can be added by the end users without having to wait for a new MobiFlight release. They simply add a new VID/PID entry to the appropriate board definition file and it will start working.
4. It sets up a framework for adding additional non-Arduino boards in the future. For example it's trivial now to know whether a board supports being flashed, and if not just skip the firmware update checks.
5. Atmega-based boards that are full custom (like my De-Ice Panel) only need an XML file added to the board definition folder to be recognized by MobiFlight.

What changes in the code

1. A new BoardManager class is created that knows how to load board definitions and search for matching board definitions based on FriendlyName, VID/PID, and MobiFlightType.
2. A new Board class is created that contains properties for all board-specific options.
3. MobiFlightCache.cs:getArduinoPorts() maintains the same general logic but the code simplifies considerably and uses calls to BoardManager to determine whether a connected device is valid for use with MobiFlight.
4. MobiFlightCache.cs:LookupAllConnectedArduinoModules() simplifies a bit as it no longer needs to do specific checks when the board doesn't have the MF firmware installed.
5. MobiFlightModule.cs:GetInfo() explicitly looks up the board definition based on the reported type from the firmware and stores it on the module, falling back to the type detected via FriendlyName or VID/PID if no firmware is installed.
6. MobiFlightModuleConfig.cs is deleted as the class is no longer used anywhere.
7. MobiFlightModuleInfo.cs simplifies down to just basic info about the connection for the device, as all the other information now comes from the Board class.
8. Anywhere in the code that used to test for board type now uses explicit properties on the Board class. This winds up touching many places in the code but doesn't change actual behaviour. It's just replacing checks against an old hardcoded list with new ones from Board properties.

A branch with these changes is available on GitHub. It works, both with current boards and new boards like the Nano or my custom ATmega32u4 board. The one thing I'm still not sure about is why MobiFlightModuleInfo.cs is still needed, but I haven't taken the time to dig into where in the code it gets really used to see if it can be refactored away somehow.

[neilenns]

Example of a board definition file: https://github.dev/neilenns/MobiFlight-Connector/blob/532805f39aeaa56e3b005da9a4993b4b513b90cc/Boards/arduino_mega.board.xml#L1

[elral]

I thought severel times about this topic and if I should answer as I have to less knowledge on the connector side.
For me it's a good step for future use to support additional HW on the firmware side, maybe not even Atmel processors (in this case one have to think how to intergrate different upload tools). Such a flexible way as proposed would make it happen, but I can not judge what the best way would be to integrate this into the Connector.
I hope that I will find some time in the next days to give this a try.

/edit: not to forget, an editor for board xml files would be much helpfull
/2nd edit: identiying a board via it's in the firmware hardcoded type would be also my approach

[neilenns]

The XML file is pretty straightforward so I'm not convinced a dedicated editor beyond the auto-complete provided by VSCode is necessary. If we wanted to get fancy we could define an XML schema and I bet VSCode would enforce/validate it automatically.

With regards to different upload tools the way I structured my proposed changes would directly make that possible. I made slight naming changes to the existing firmware update functions so it's clear they are avrdude specific, and using the XML to provide the parameters and select avrdude as the method of update.

[MobiFlight-Admin]

Hey Neill! Thanks for all the great write up and effort! In general, i had thought about this in the past and I am happy to see that we are now taking a stab at this.

There are some concerns that I would like to bring up:

• If people can start adding there board definitions easily we might see more support requests for exotic boards, which requires more bandwidth to from the community
• If people fix a board support by adding a VID PID on their end, we might never get to know about it. Right now people have to report it so that it gets added to the list - which is positive for everyone

There should be an official repository then for boards probably and a community repository.

I will have a look at all the changes later today - in general I am supporting this functionality a lot.

[neilenns]

The official repository is easy: it's GitHub, and the supported board definition files ship included with MobiFlight on every release.

For exotic boards the support really comes down to three things:

1. Can MobiFlight detect the board and know it is valid to use?
2. Is there firmware that runs on the device?
3. Can MobiFlight load the firmware onto the device?

The nice thing about these changes is board support can be done at any of those three levels. For @elral66 and me, our own custom board XML files to get level 1 support is a huge win. We know how to manually load firmware on other devices and don't care about MobiFlight doing it for us. We just want MobiFlight to recognize the board and know the pin assignments.

For non-Arduino boards support could be done at level 2 without direct MobiFlight involvement as well. Teensy support (a logical next one for example) could be done by @elral66 providing his build of the Teensy firmware for people to use and the accompanying board definition file. Or you could imagine someone selling a pre-build FMC and providing the firmware on it already and the board definition file to put in the right folder to make it work with MobiFlight.

For full support the additional work would simply be adding a new method to the firmware update code to know how to use a different command line tool to use a different flashing tool. I structured the changes so this is straightforward to do in the future.

[neilenns]

With regards to fixing VID/PID, my guess is people will still report to us via Discord or forum. We just have to be mindful that any time someone reports it a bug gets logged in addition to giving a temporary workaround.

[elral]

Not sure about the support topic, it could make thinges easier, but it could vice versa. I would expect that a user how creates his own support file should be able to figure out things that are going wrong, espacially if he set's up his own board definition in platformio.ini. But I might be wrong that users will give "every" board he has a try.
The support topic in the forum and in discord could be limited to "official" supported boards, that might help a little bit.
On the other new HW could be easy "implemented", might be a big win for experienced user and can be profite from.

Regarding the Teensy I am not sure if this would be at least a first step. The Teensy35 with 5V tolerant inputs would be good, but the I2C is not running in this case. So even for the Teensy35 a level shifter is required which would be needed for all new HW (not Atmel familiy). This can be rise to the biggest support topic. And the Teensy familiy is quite expensive.
The Rasperry Pico is a really nice board, the formfactor is a littel bit bigger than the ProMicro, has some more I/O's, lots of flash and RAM and at least cost only ~5€.
But this is not a question which has to be answered soon.

[neilenns]

Opened a draft PR for this. I found a few places I need to make some tweaks while doing so, plus need to update the unit tests.

[MobiFlight-Admin]

Let's go JSON

{
    "Board" : {
        "Info" : {
            "LatestFirmwareVersion" : "1.11.3",
            "FriendlyName" : "Arduino Uno",
            "MobiFlightType" : "MobiFlight Uno",
            "CanInstallFirmware" : false,
            "ForceResetOnFirmwareUpdate" : false
        },
        "Capabilities" : {
            "Outputs" : 8,
            "Buttons" : 8,
            "LedSegments" : 1,
            "Encoders" : 2,
            "Steppers" : 2,
            "LcdI2C" : 2,
            "Analog" : 2
        },
        "Connection" : {
            "BaudRate" :  115200,
            "ConnectionDelay" : 1750,
            "DtrEnable" : true,
            "ExtraConnectionRetry" : true,
            "MessageSize" : 32,
            "EEPROMSize" : 256
        },
        "HardwareIds" : [
            "^VID_2341&PID_0043",
            "^VID_2341&PID_0043",
            "^VID_2341&PID_0043",
            "^VID_2341&PID_0043"
        ],
        "AvrDudeSettings" : {
            "Device" : "atmega328p",
            "BaudRate" :  115200,
            "Programmer" : "arduino",
            "FirmwareBaseName" : "mobiflight_uno"
        },
        "Pins" : [
            { "Pin": "2", "isAnalog" : false, "isPWM" : false, "isI2C" : false },
            { "Pin": "3", "isAnalog" : false, "isPWM" : true, "isI2C" : false },
            { "Pin": "4", "isAnalog" : false, "isPWM" : false, "isI2C" : false },
            { "Pin": "5", "isAnalog" : false, "isPWM" : true, "isI2C" : false },
            { "Pin": "6", "isAnalog" : false, "isPWM" : true, "isI2C" : false },
            { "Pin": "7", "isAnalog" : false, "isPWM" : false, "isI2C" : false },
            { "Pin": "8", "isAnalog" : false, "isPWM" : false, "isI2C" : false },
            { "Pin": "9", "isAnalog" : false, "isPWM" : true, "isI2C" : false },
            { "Pin":"10", "isAnalog" : false, "isPWM" : true, "isI2C" : false },
            { "Pin":"11", "isAnalog" : false, "isPWM" : true, "isI2C" : false },
            { "Pin":"12", "isAnalog" : false, "isPWM" : false, "isI2C" : false },
            { "Pin":"13", "isAnalog" : false, "isPWM" : false, "isI2C" : false },
            { "Pin":"14", "isAnalog" : true,  "isPWM" : false, "isI2C" : false, "Name" : "A0" },
            { "Pin":"15", "isAnalog" : true,  "isPWM" : false, "isI2C" : false, "Name" : "A1" },
            { "Pin":"16", "isAnalog" : true,  "isPWM" : false, "isI2C" : false, "Name" : "A2" },
            { "Pin":"17", "isAnalog" : true,  "isPWM" : false, "isI2C" : false, "Name" : "A3" },
            { "Pin":"18", "isAnalog" : true,  "isPWM" : false, "isI2C" : true, "Name" : "A4" },
            { "Pin":"19", "isAnalog" : true,  "isPWM" : false, "isI2C" : true, "Name" : "A5" }
        ]
    }
}

[MobiFlight-Admin]

It looks so neat!

[neilenns]

Yeah but it's a whole new library to pull in and start using. Other than looking neat, XML is how everything else in MobiFlight reads/writes things to files.

[neilenns]

Ok I made it JSON. And I added a JSON schema file for good measure, gives full Intellisense support in Visual Studio Code.

[MobiFlight-Admin]

that's super cool!

[MobiFlight-Admin]

i am so excited to check later in the evening

[elral]

It would be good directly to think about how different flash tools.
I guess something like:
"FlashTool" : { "Tool" : "avrdude" },
should be added.
Flash tools for other HW are always available and should be stored similiar to the existing folder structure.

A topic could be special procedures to get into the bootloader like the promicro (which also applies to e.g. the Raspberry Pico).

[neilenns]

I have different flash tool support done in the PR. Each flash tool gets its own block of settings in the board definition file, and the presence of that block determines the flash tool to use. Right now the only supported tool is avrdude but the code is structured to make it straightforward to add a different one.

For the "toggle com port to get into bootloader" I made a dedicated, generic, setting for that which controls the behaviour. So a Raspberry Pico would just set that flag in the board config file for the pico and would get the same behaviour prior to flashing as the Pro Micro.

[DocMoebiuz]

This has been implemented and is working great! Thanks @neilenns and everyone else who helped!